Murphy's Law
In a recent conversation I was graciously reminded that sometime between 1948 and 1949 following a mishap during rocket sled tests, American aerospace engineer Edward A. Murphy Jr. uttered this precautionary design advice:
If there are two or more ways to do something and one of those results in a catastrophe, then someone will do it that way.
Edward A. Murphy Jr.
"Murphy's law", as it has become known, is an adage or epigram that is typically stated as: "Anything that can go wrong will go wrong." In some formulations, it is extended to "Anything that can go wrong will go wrong, and at the worst possible time."
There's no better or worst time for things to go wrong than right at project launch when an unexpected resource allocation issue delays establishing a development environment.
When I started The Laundry King project, I promised to share the good, the bad, and the ugly. Never did I expect that the client's host didn't have the room to create a pristine development environment. It's never a good idea to develop anything in production, so the work around is to host the new development environment on my dedicated server.
It sound's simple enough, right? All I really need is the Formidable Forms and data. I could create a pristine WordPress environment with only the necessary Formidable Forms add-ons and export/import the data I need to populate the development environment. That was the plan anyway.
The Data
The Laundry King project targets 5 main forms consisting of 1,152 fields. One of the main forms has 4 embedded repeater forms. These 9 forms hold 253,513 entries.
| Form | Fields | Entries |
|---|---|---|
| Locations | 24 | 88 |
| Soap King FIF | 849 | 2,134 |
| Soap King Inventory Data | 3 | 220 |
| Soap King Order Form | 40 | 10,108 (parent) 239,592 (repeaters) |
| Store Visitation Report | 252 | 1,371 |
| Totals | 1,152 | 253,513 |
Importing the first three forms was no issue. Locations, Soap King Inventory Data, and the Store Visitation Report exported fine from the production system as XML and imported into the development environment quite smoothly, but the other two forms were problematic.
Problem 1: Broken Export XML
Neither the Soap King FIF nor Soap King Order Form could be imported as XML if the data was included in the export. Formidable appeared to output the files correctly from production, but as soon as I tried importing them into development, Formidable's import process failed with a malformed XML error. Something was wrong in the data. If I did not include the data, the forms imported perfectly.
This is where the real problems started and after significant troubleshooting and help from a Strategy 11 support specialist, I had to develop a work around.
Problem 2: Notice: ob_end_flush()
Because I could not get the data to import through XML, there was no choice but to try to import it from CSV files. I started with the Soap King FIF data. That form only has 2,134 entries and should have imported without an issue, but the CSV import would only process the first 502 records and then it would error with this notice:
Notice: ob_end_flush(): Failed to send buffer of zlib output compression (0)
This message is not generated by Formidable, this is a WordPress message from wp-includes/functions.php.
Buffer errors are usually due to PHP memory configuration constraints. This form has 849 fields and processing that much data is likely the root cause of the problem. Googling this notice produced recommendations to increase PHP memory, which I did to 2048M. It turned out not to be the solution, so it was time to open a ticket with Strategy 11.
Kolahn at Strategy 11 support successfully imported the data on his Cloudways VULTR based server requiring only a minor configuration change for the PHP memory limit. Our dedicated server still runs CentOS and PHP 8.2. It may be possible that the deprecated CentOS may be at the heart of the issue.
The work around is to reduce the CSV import files to no more than 400 entries each. There are only two ways to do this to a Formidable export file.
- You either have to painstakingly use an editor to save 400 entries at a time manually with the header record, OR,
- You write a tool to do it for you. I opted to write a tool and to get started I asked ChatGPT to provide sample code to refresh my memory on low-level file operations.
For the first time ever in my experience, ChatGPT wrote a function that worked without modification in repsonse to this question:
Write a PHP function to load a large CSV file and loop through it to create smaller files of 400 rows each
And it works better than I imagined at splitting large CSV files into manageable chunks. Here's the public gist I created to share the function. This is one you should save in your CSV toolbox.
In less than a few seconds, this code split the 2,134 CSV entries into 6 files and all loaded without further incident.
Problem 3: Store King Order Form
This is the form with 40 fields, four of which are repeaters. A repeater field is in fact not a field. It is an example of an embedded form that allows you to add multiple form entries into a child form from its parent's user interface.
If you'll also recall, this form has 10,108 (parent) and 239,592 (repeater) entries. Normally, Formidable imports just work. Before splitting this CSV file into chunks, I thought I'd try a full import first.
This import did better than the FIF form, but the ob_end_flush() error raised it's ugly head once again causing the import to fail after about 1,500 entries. So, once again, this CSV file needed to be chunked into 500 entry files, 21 of them to be exact.
Why did 225 entries disappear?
Paranormal?
Not likely! But even with this additional effort, something strange happened. In the production system, the form has 10,333 parent records. After the import completed, the development system had 10,108 parent records.
There were no errors captured during the import of all of the chunked files. The entries simply vanished! There will be no investigations to discover why either.
This is a development environment where I need as complete a set of representative data as possible. It doesn't have to be exact. So for now, the saga of missing entries after a supposedly clean import will have to wait for another day, and quite possibly, for another ghost hunter.
How Formidable Structures CSV Repeater Exports
There's always something new to learn about how Formidable works. This is the first time I ever needed to export a form to CSV that included repeaters. The structure of the CSV file is intriguing. Every field in a repeater entry receives its own set of columns. So if a repeater has 4-fields and there are 10-repeater entries, there will be 10 x 4 columns added to the parent entry's CSV output.
What else I find interesting is that Formidable must be calculating the maximum number of entries each repeater has before creating the export file so it can structure the header properly. It will create a CSV file with 4 x max-entries columns added to the export.
In Excel, the header length of the export file reached column MZ and that means each row contained 364 data columns from a form that only has 40 fields.
Problem 4: Relink Dynamic Fields
This problem is reserved for the next Developers Corner article.
Leave a Reply