Cause

Recently, we have been pushing forward the modularization of app components, but starting from a certain day, Jenkins on the build machine could no longer archive successfully. The error code returned by Jenkins was 65. Although it was an error, there was no error log at all, and it just mysteriously ended with Build Failed.

Verify final result code for completed build operation
Build operation failed without specifying any errors. Individual build tasks may have failed for unknown reasons.
One possible cause is if there are too many (possibly zombie) processes; in this case, rebooting may fix the problem.
Some individual build task failures (up to 12) may be listed below.

Since it seemed related to component packages, we first tried rolling back the code. However, quite a long time had passed since the last successful package build, so a large number of merge requests had already been merged into the main branch. After a series of attempts, packaging still failed, so we gave up on that approach.

Later, we found that archiving could succeed on our own computers, but not on the build machines (neither Mac mini nor Mac Pro worked). The build machines inject an enterprise certificate for packaging and release, while locally we use a normal company certificate. So we guessed that the enterprise certificate might be the cause. After comparing for a long time, we found a .mobileprovision file in the project directory. Surprisingly, after deleting it, archive succeeded! But reality poured cold water on us again: the second build still failed. It now seemed that this error occurred probabilistically, and with a very high probability.

Helplessly, we continued reading the logs. After removing other unrelated items in Build Phases from the project, we found that deleting [CP] Copy Pods Resources allowed the package to build successfully, while keeping it caused failure. After understanding the structure of this part, we guessed that the problem might be caused by too many files in Input Files and Output Files at this stage. To verify our guess, we deleted the resource directories in the split-out components, and sure enough, the build succeeded.

Analysis

During the modularization process, in order to simplify the splitting work, we only set resource in the podspec as the resource index directory, instead of using resource_bundles recommended by cocoapods as the carrier for resource files (because that would require too many changes to the original code and would be error-prone). In this way, when cocoapods runs pod install, all resources are copied directly into the main project. And cocoapods implements this by adding the [CP] Copy Pods Resources phase in Build Phases, where all resource files in the Pod are added, causing the number of files in this phase to become extremely large. Such a huge quantity may trigger some bug in Xcode, leading to packaging failure. Therefore, one temporary solution is to move the resource files from Pods into the main project during packaging.

Solution

The code is written in Ruby and uses the Xcodeproj project, as follows.

#!/usr/bin/ruby

require 'xcodeproj'
require 'fileutils'

def add_file_resources(direc, current_group, main_target)
    Dir.glob(direc) do |item|
        next if item == '.' or item == ".." or item == '.DS_Store'
        isAsset = item.include? ".xcassets"
        if File.directory?(item) and !isAsset
            new_folder = File.basename(item)
            created_group = current_group.new_group(new_folder)
            add_file_resources("#{item}/*", created_group, main_target)
        else 
            # i = current_group.new_file(item)
            i = current_group.new_reference(item)
            puts "[Project] add: #{item}"
            # main_target.add_file_references([i])
            main_target.add_resources([i])
        end
    end
end

def move_assets_folder(path, depth)  
    Dir.entries(path).each do |sub| 
        next if sub == '.' or sub == '..'  
        if File.directory?("#{path}/#{sub}")  
            next if depth > 3
            if "#{sub}" == "Assets"
                current_project = path.split('/').last
                FileUtils.mv "#{path}/#{sub}", "./podsAsset/#{current_project}"
                puts "[Folder] #{path}/#{sub} => podsAsset/#{current_project}"
            else
                move_assets_folder("#{path}/#{sub}", depth + 1)  
            end
        end  
    end  
end

if(File.exist?("podsAsset"))
    puts "[Folder] 'podsAsset' already exist, remove it!"
    FileUtils.rm_rf("podsAsset")
end
FileUtils.mkdir_p("podsAsset")

move_assets_folder("modules", 1)

project = Xcodeproj::Project.open("./xxx.xcodeproj")
target = project.targets.first

group = project.new_group('podsAsset')
group.set_source_tree('SOURCE_ROOT')
add_file_resources("podsAsset/*", group, target)
project.save

puts "[Done]"

Some of our components that are more closely tied to the project are placed directly in the modules folder under the root directory, so as long as the resource files of the components in this directory are copied into the main project, there should be no problem. The specific idea is to traverse the modules folder in the current directory to look for the Assets folder (as required by the modularization convention). If found, it is moved to the podsAsset folder under the root directory. In order to accurately find the correct Assets folder, I set a maximum search depth of 3. After traversing all directories, I then call the relevant methods in Xcodeproj to add all files in those directories into the project.

Execute this Ruby script before packaging the project, and after setting the certificate, the package builds successfully.

Postscript

At this point, the problem is basically solved. However, this can only be considered a temporary solution. We still need to further discuss how resource files should be organized in the modularization process. If possible, new components should still be placed in an independent bundle.

In addition, there is still one unresolved question. Why do some machines fail to archive, while this problem does not appear on our development machines? If any of you have a deeper understanding of this issue or a more elegant solution, please let us know!