/dev/trouble
Eric Roller's Development Blog

Framing iPhone 14 screenshots

- Posted in iOS by

When trying to place generated screenshots into device frames (automated by fastlane's frameit), I discovered that, since the iPhone 14 devices have small bezels, the corners of the screenshot can be seen spilling over to outside of the template frames for these devices:

trim_devices = [
    "iPhone 14 Plus",
    "iPhone 14 Pro",
    "iPhone 14 Pro Max",
]

I solved this by cutting the corners off of the screenshot images before running frameit. This can be done by three ImageMagick commands, creating a rounded-rect image with 50-pixel-radius curved corners:

`convert '#{file}' 
    -format 'roundrectangle 1,1 %[fx:w+4],%[fx:h+4] 50,50' 
    info: > mask.mvg`
`convert '#{file}' 
    -alpha transparent 
    -background none 
    -fill white 
    -stroke none 
    -strokewidth 0 
    -draw "@mask.mvg" mask.png`
`convert '#{file}' 
    -alpha set 
    -bordercolor none 
    mask.png 
    -compose DstIn 
    -composite '#{file_name}'`

In my setup, I am operating in a temporary directory (with a language subdirectory, e.g. "en-US"):

Dir.mktmpdir do |tmp|
  path_temp = File.join(tmp, language)
  Dir.mkdir(path_temp)
  puts "Operating in #{path_temp}"

  num_found = 0
  # ...(see below)...
end

Then for each of the devices, depending on whether it is a large iPhone 14 device or not, I either create a corner-cropped image in the temporary directory or place a copy of - or rather a symbolic link to - the original image:

path_shot = File.absolute_path(File.join(
      "..", output_directory, language
  ))

run_devices.each do |device|
  Dir.glob(File.join(path_shot, device + "-*.png")).each do |file|
    file_name = File.basename(file)
    Dir.chdir(path_temp) do
      if trim_devices.include? device
        # Cut off rounded corners using ImageMagick:
        puts "Rounded frame for: #{file_name}"
        `convert '#{file}' -format 'roundrectangle 1,1 %[fx:w+4],%[fx:h+4] 50,50' info: > mask.mvg`
        `convert '#{file}' -alpha transparent -background none -fill white -stroke none -strokewidth 0 -draw "@mask.mvg" mask.png`
        `convert '#{file}' -alpha set -bordercolor none mask.png -compose DstIn -composite '#{file_name}'`
        Dir.glob("mask.*").each { |mask| File.delete(mask) }
      else
        puts "Linking frame for: #{file_name}"
        File.symlink(file, file_name)
      end
    end
    num_found += 1
  end
end

After adding Framefile.json as well as any other needed files into the temporary directory, frameit can be called to do its work there:

frame_screenshots(
    use_platform: "IOS",
    path: tmp
)

Thereafter the "*_framed.png" images need to be copied before the temporary directory is deleted. Here is an example where the result frames are placed in a separate "en-US_framed" directory:

path_fram = File.absolute_path(File.join(
    "..", output_directory, language + "_framed"
))

Dir.glob(File.join(path_temp, "*_framed.png")).each do |file|
  file_name = File.basename(file).sub(/_framed/, "")
  puts "Frame generated: #{file_name}"
  FileUtils.cp(file, File.join(path_fram, file_name))
end

In my setup, fastlane usually runs on a separate computer in my closet. Therefore I find it helpful to add audio messages such that I can hear the status:

sh 'say "Creating screenshots for iOS devices."'

Fastlane's say() command sometimes hasn't worked for me. That's why I have used sh 'say'. However, that was recorded by fastlane as an action. Using this Ruby syntax is not tracked as an action:

`say "Processing images."`

But neither does its produce any text in the log. So I created this Ruby function which prints the message and then speaks it:

def talk(s) puts s; `say "#{s}"` end

With the function anywhere in fastlanes Fastfile, one can just call it like a command:

talk "Framing screenshots"

Speaking Multiple Phrases

This all works nicely, but I discovered that my error message needs to be made more prominent. I might be in the bathroom and don't hear the announcement. That's why I have now changed it to be a lot longer and a lot more annoying.

error do |lane, exception|
  phrases = [
      "Are you serious?",
      "Huston, we've had a problem!",
      "I can't believe it!",
      "I thought you had fixed that.",
      "Is it broken again?",
      "Maybe we can just reboot it?",
      "No! No! No, not again!",
      "Oops! I did it again!",
      "Something smells fishy!",
      "There is a left-over screw here.",
      "There must be a rational explanation for this.",
      "This must be a mistake!",
      "What if we pressed the red button?",
      "What is going on?",
      "What is this switch for?",
      "Where's the beef?",
      "Where is the user manual?",
      "Who pressed the red button?",
      "Who let the dogs out?",
      "You must be joking!",
  ]

  `say "#{ phrases.shuffle.join(" [[slnc 2500]] ") }"`
end

Note that the phrases are shuffled and concatenated by [[sinc 2500]] instructions which add 2.5-second-long pauses between the phrases. I found that in Apple's Speech Synthesis Programming Guide.

Creating screenshots is a tedious task that lends itself to automation. Ideally, it would be preferrable if it were possible to have simple, device-idependent testing scripts. The reality is that many exceptions creep into the code that simulates the UI interactions when generating the screenshots:

if UIDevice.current.userInterfaceIdiom == .phone {
    // Do something different for iPhones.
}

But the real problem is maintaining these scripts. Whether it is iOS SDK changes, new device names, new screen sizes, or indeed new apps features, suddenly the scripts no longer work. Even if everything worked nicely last time, the tiniest of changes will trigger hours of UI scripts re-tuning.

I usually find myself having to re-run fastlane over and over in order to create new screen shots. Quite often, the scripts work well for some devices but fail for others. At that point it would be nice if fastlane could be instructed to only re-run the scripts for the devices that had failed. But fastlane has no option for that.

But it is possible to re-code the Fastfile since it is essentially a Ruby script. All I need is a simple check whether the final screenshot exists for a device. When present, don't re-run the lengthy UI scripts for that device.

So here are the relevant parts of my new ./fastlane/Fastfile; beware custom names like MyApp:

delete_all_screenshots = false

swift_macro_defs =
    "SWIFT_ACTIVE_COMPILATION_CONDITIONS='FASTLANE'"
output_directory = "./screenshots/fastlane"

# All languages and devices to run:
all_languages = [ "en-us" ]

all_devices = [
    "iPad Pro (9.7-inch)",
    "iPad Pro (10.5-inch)",
##  "iPad Pro (11-inch)",
    "iPad Pro (11-inch) (4th generation)",
    "iPad Pro (12.9-inch) (2nd generation)",
    "iPad Pro (12.9-inch) (6th generation)",
    "iPhone 8",
    "iPhone 8 Plus",
    "iPhone 14",
    "iPhone 14 Plus",
    "iPhone 14 Pro",
    "iPhone 14 Pro Max",
    "Apple TV",
    "Apple TV 4K (3rd generation)",
]

platform ::ios do
  desc "Take screenshots (iOS)"
  lane :ios_screenshot do

    all_languages.each do |language|
      # Work out the path to the screenshots.
      # Beware: CWD is the directory of this file!
      path_shot = File.join(
          "..", output_directory, language
      )

      run_devices = Array.new

      all_devices.each do |device|
        # Neither 'Apple TV' nor 'Apple Watch' devices
        next if device.start_with? "Apple"

        unless delete_all_screenshots then
          # If the last screenshot exists, skip this device:
          last_shot = device + "-10_Exercises.lite.png"
          last_shot = device + "-09_Exercises.lite.png" \
              if device.start_with? "iPad"

##        puts "Checking if file exists: #{last_shot}"
          file_shot = File.join(path_shot, last_shot)
          next if File.exists?(file_shot)
        end

        run_devices << device
      end

      next if run_devices.empty?

      snapshot(
          project: "./MyApp.xcodeproj",
          scheme: "MyApp-iOS",
          xcargs: swift_macro_defs,
          devices: run_devices,
          languages: [ language ],
          localize_simulator: true,
          reinstall_app: true,
          headless: false,
          concurrent_simulators: false,
          clean: delete_all_screenshots,
          clear_previous_screenshots:
              delete_all_screenshots,
          stop_after_first_error: true,
          number_of_retries: 0,
          override_status_bar: true,
          test_without_building: false,
          output_directory: output_directory,
          disable_slide_to_type: true,
          skip_open_summary: true,
          only_testing: [
              "UITests-iOD/UITests_Fastlane",
          ],
      )

      delete_all_screenshots = false
    end
  end
end

If you want to recreate all screenshots, you could set delete_all_screenshots to true or, well, delete the files manually.

Also note the SWIFT_ACTIVE_COMPILATION_CONDITIONS compiler flag. This allows me to define a FASTLANE macro that can be used in the app's Swift code:

#if FASTLANE
loadExampleData()
#endif

For a long time, I have been trying to avoid scroll bars in the screenshots that I generate. The solution that I came up with was simply to wait for them to disappear:

thirdBlock.element.swipeUp()

// Wait for the scroll indicators to be hidden
Thread.sleep(forTimeInterval: 2.5)

takeSnapshot("Last_Block")

I have also been using fastlane for several years to automate this. While I try very hard never to add any screenshot-specific exceptions to the code, there has always been the odd need to do it, for instance to handle situations that are not supported by the iOS simulator.

Preprocessor Macros

For code sections that are only used during development, you should mask them out using "preprocessor macros" like:

#if DEBUG
print("Data finished loading.")
#endif

This works using an Xcode project setting that defines SWIFT_ACTIVE_COMPILATION_CONDITIONS to use DEBUG, which results in the command-line option -D DEBUG to be used by CompileSwift. You can see this if you dive deep into your build logs.

Similarly, you can mask sections that are only (or never) to be used in the simulator like this:

#if targetEnvironment(simulator)
    // iOS simulator only
#else
    // iOS device only
#endif

Simple inversions are also supported in Swift using an exclamation mark (read it as: "not"):

#if ! targetEnvironment(simulator)
    // iOS device only
#endif

Fastlane-Specific Code

The same can be done for fastlane-specific code, but it is necessary to define a macro, for instance in the fastlane/Snapfile. Here we define FASTLANE:

# Add a define for xcodebuild
xcargs "SWIFT_ACTIVE_COMPILATION_CONDITIONS=FASTLANE"

which allow us to use in the code:

#if FASTLANE
    // fastlane only
#endif

No Scroll Bars in Fastlane

Using the above macro, I have now simply disabled the scroll bar indicators in the code:

override func viewDidLoad() {
    super.viewDidLoad()

    navigationItem.rightBarButtonItem = editButtonItem

    #if FASTLANE
    tableView.showsVerticalScrollIndicator = false
    tableView.bounces = false
    #endif
}

And this means that the test script can be made to run without additional delays (compare above):

thirdBlock.element.swipeUp()
takeSnapshot("Last_Block")

[UPDATE 2021-01-21] In SwiftUI, you could use:

List {
    ...
}
.onAppear {
    #if FASTLANE
    UITableView.appearance()
        .showsVerticalScrollIndicator = false
    #endif
}

When creating screenshots with fastlane, there is support for checking at runtime within the app whether fastlane is being used by looking at a user default:

if UserDefaults.standard.bool(forKey: "FASTLANE_SNAPSHOT") {
    // runtime check that we are in snapshot mode
}

Unfortunately, that does not work from within the test code since the tests are run in a separate helper app!

In the test code, I would like to use something like the following such that I can test and tweak the code within Xcode before running fastlane on it:

#if FASTLANE
    snapshot("1_init", timeWaitingForIdle: 0)
#else
    let attach = XCTAttachment(screenshot: XCUIScreen.main.screenshot())
    attach.lifetime = .keepAlways
    add(attach)
#endif

One would think the FASTLANE_SNAPSHOT=YES build setting would be suitable for this task, but I have not found any way to detect it from within Swift, possibly because values like YES are not supported by the compiler.

This, however, seems to do the trick: Add this line to your Snapfile:

# Add a define for xcodebuild
xcargs "SWIFT_ACTIVE_COMPILATION_CONDITIONS=FASTLANE"