I had fun writing about how I work with Git yesterday. I thought I’d continue on that thread.

I have a solid set of code libraries that I’ve written that latch into the WordPress themes we produce at iThemes. Each time code is duplicated across different repositories, I break that code out and make it into a separate repository. I then link it back into the project as a submodule. This makes it extremely-easy to keep duplicated code across numerous repositories updated with little or no fuss.

After cloning a repository, simply run git submodule init followed by git submodule update in order to initialize all the submodules and update their container folder with the content of the submodule’s repository. For a long time, this is exactly what I did when I would clone a theme repository to start working on it. However, this quickly wasn’t enough.

The problem happened as soon as I added a submodule to a repository that was also a submodule of other repositories. Doing the submodule init and update process wouldn’t do everything I needed in this case as there would be submodules in some subfolder that haven’t been set up.

I didn’t want to get into a habit of always switching to other directories and doing the submodule processes there as well since I 1) knew that I would forget all-too-often, thus wasting my time, and 2) knew that this would not be the last time that a submodule had submodules. Heck, there is even the possibility that I’ll have a submodule that has a submodule that has a submodule. It was immediately clear that I needed a script to do all this dirty work for me. The rest of this post will be about the script I created.

The Script

First, I’ll share the script itself. If you are interested in how it works, continue reading.

git-submodule-recursive-update (right-click > “Save Link As…” to download)

The script is written in Perl and should work on most systems. I’ve only tested it on Linux and OS X, so please let me know your results if you try out on Windows.

#!/usr/bin/perl

use strict;
use Cwd;

init_and_update();

exit;

sub init_and_update
{
    my $start_path = cwd();

    my %paths;
    my $updated;

    do
    {
        my $data = `find . -name '.gitmodules'`;
        chomp($data);

        $data =~ s/\/\.gitmodules//g;

        foreach my $path (split(/\n/, $data))
        {
            $paths{$path} = '' if($paths{$path} eq '');
        }

        $updated = 0;

        foreach my $path (sort keys %paths)
        {
            if($paths{$path} eq '')
            {
                chdir($path);
                `git submodule init 2>&1`;
                `git submodule update 2>&1`;
                `git submodule foreach 'git checkout master' 2>&1`
                chdir($start_path);

                if($ARGV[0] eq '--remove-gitmodules')
                {
                    unlink("$path/.gitmodules");
                }

                $paths{$path} = 1;

                $updated++;
            }
        }
    } while($updated);
}

The Description

The functionality should look very straight-forward to anyone that knows Perl.

1. I store the current directory in the $start_path variable in order to always know where home is.
2. I start a wrapper loop that keeps running until all the possible submodules are initialized and updated.
3. Using the find command, I look for all the instances of .gitmodules and store the results in $data. The .gitmodules file exists if a repo has submodules.
4. I remove all the .gitmodules file references from the $data to leave just the paths.
5. I split the paths into an array and initialize the %paths hash to have a blank value for new paths (stored in the key). Setting this value to blank will flag the following loop that the submodules in that path have not been set up yet.
6. I create a tracker variable, $updated, to check if anything happened in the loop.
7. I then loop through the %paths hash to work on each path. If the path’s hash value is blank, I process that path.
8. I cd into the repo path, init and update the submodules, and switch back to the starting folder.
9. If the script is called with the optional --remove-gitmodules argument, I remove the .gitmodules folder while I’m focused on that folder. I use this for other automation scripts, so it may or may not be of value to you.
10. I then set the path’s hash value to 1 to flag it as done.
11. Closing out the loop, I update the $updated variable to show that something was updated in this pass.
12. Finishing up the do loop towards the top, I have while($updated). Basically, as long as something was updated in the core update loop, I’ll run everything again. This means that the loop will keep running until it didn’t find anything else that needed to be updated. When that point is reached, the main loop ends, and the script is finished.

I know that there are a number of things I could have done to make for a much more brief, compact script, but I was going to quick production with solid functionality, not brevity. In addition, there are unnecessary elements such as incrementing the $updated variable rather than just setting it to some value. I thought might want to know how many things were updated at some point, so I left it as a counter.

If you found this script helpful, please leave a comment. The more interest these Git-related posts receive, the more motivated I’ll be to share other processes, developments I’ve made to make working Git easier.

I work for a company that has a proprietary CMS software package written in Perl. We have a number of servers that run this code, and everything has been fine for a number of years and many different versions and customizations.  A few months back, something changed. One of our servers started producing completely empty files for all uploads. This affected all of our code, all of our versions, and every site.

After some time debugging and testing, I finally found the problem. Apparently some code was updated, either Perl itself or one of its packages, and that caused my CGI object to be recylced before the upload code ran. When the CGI object gets recylced, all the file handles are closed resulting in reading and saving an empty file.

The solution was deceptively simple. All I had to do was store the CGI object in a persistent variable that has scope throughout the end of the program execution.