如何在示例后挂钩具有执行结果和状态：aggregated_failures标志

Question

我正在维护一个用Rspec＆Capybara和SitePrism（No Rails）编写的独立测试自动化套件。

最近我开始将它与Testrail集成用于报告 - 我使用了rspec-testrail gem，但我不得不修改它，因为我还想发送带有测试进度和报告的Slack通知。

无论如何，集成工作顺利，除了示例处理依赖于具有异常和缺少异常的示例的事实导致将测试状态设置为Testrail上的Passed。看来，after :each中的after :example和Rspec.configure并不保证该示例已经完成运行。我也尝试过around :example和around :each描述here，但无济于事。

我检查了example.metadata的内容，看起来example.metadata[:execution_result]只有started_at变量，但一个完成的例子也有finished_at和status变量，根据to the docs

我怀疑（在阅读了津津有味的文档之后）是:aggregated_failures是不同元数据结构的原因，并且多个期望在稍后合并到一个回溯中的线程中运行。

你知道我怎么能等待这个例子完成或如何挂钩到它完成的状态？或者也许我应该创建一个自定义格式化程序，我会在打印到控制台的示例通知后挂钩（我想保留堆栈跟踪）。

我的代码如下：

测试（两个断言都失败）：

require 'spec_helper'

feature 'Sign in' do
  let(:login_page) { LoginPage.new }
  let(:user) { { email: ENV['ADMIN_EMAIL'], password: ENV['ADMIN_PASSWORD'] } }

  scenario 'is successful and user is redirected to dashboard for user with correct credentials', testrail_id: 5694 do
    login_page.load
    login_page.form.email.set(user[:email])
    login_page.form.password.set(user[:password])
    login_page.form.submit_btn.click
    expect(login_page.sidebar).to have_jobs(text: "some_nonexistenttext")
    login_page.logout
    expect(current_url).to have_content "google.com"
  end
end

上述测试的控制台输出：

Failures:

  1) Sign in is successful and user is redirected to dashboard for user with correct credentials
     Got 2 failures:

     1.1) Failure/Error: expect(login_page.sidebar).to have_jobs(text: "blala")
            expected #has_jobs?({:text=>"some_nonexistenttext"}) to return true, got false
          # ./spec/auth/login_spec.rb:13:in `block (3 levels) in <top (required)>'

     1.2) Failure/Error: expect(current_url).to have_content "google.com"
            expected to find text "google.com" in "https://example.com/"
          # ./spec/auth/login_spec.rb:15:in `block (3 levels) in <top (required)>'

Finished in 53.91 seconds (files took 1.45 seconds to load)
4 examples, 1 failure

Failed examples:

rspec ./spec/auth/login_spec.rb:8 # Sign in is successful and user is redirected to dashboard for user with correct credentials

规范助手：

require 'rubygems'
require 'capybara/rspec'
require 'selenium-webdriver'
require 'site_prism'
require 'slack-notifier'

require_relative '../config'
require_relative '../lib/testrail'

...

RSpec.configure do |config|
  config.define_derived_metadata do |meta|
    meta[:aggregate_failures] = true
  end
  config.example_status_persistence_file_path = 'examples.txt'
  config.before :all do
    testrail_initialize_test_run!
  end

  config.after :example, testrail_id: proc { |value| !value.nil? } do |example|
    RSpec::Testrail.process(example)
  end
end

处理方法（slightly modified from original）

  def process(example)
    if example.exception
      status = 5
      message = example.exception.message
      slack_notifier.publish(message_text "Failed")
    elsif example.skipped? || example.pending?
      puts 'Example skipped or pending'
      status = 10
      message = 'Pending or not implemented'
    else
      status = 1
      message = ''
    end
    client.send_post("add_result_for_case/#{testrun['id']}/#{example.metadata[:testrail_id]}",
                      status_id: status,
                      comment: message)
  end

Answer 1

所以我基本上只需要使用记者监听器并处理其中的通知:)

config.reporter.register_listener RSpec::Testrail::Listener.new, :start, :example_failed, :example_passed, :example_pending, :stop