eZ Platform Discussions

Merge and upgrade several databases


#1

Hello to all

I have a new migration to do. It’s more a redesign actually because we do not keep the code.

But I have to recover the contents.

The thing is that the source site is a multi-site in eZ4.0.1 with a database per site. And that the target is a eZPlatform Enterprise multi-site with a single database!

What strategy do you recommend to recover this data?


#2

I don’t think you have much options here.
Pick the biggest and most complex site and upgrade it and create the new main intance, then import the data from other sites:

  • either with content export to packages then importing them to the main instance
  • or by making a custom script which reads data from other sites and syncs them to the main instance

#3

Kaliop Migrations bundle could be useful too.


#4

I have 5 sites with more than 4000 items to retrieve in each. (And I do not know yet if I will have to recover the other types of contents …)

Of course, when the new multisite goes into production, I’ll be able to play the last sync quickly so I do not have to block the contribution too long …

The idea of going on a “migration” seemed pleasant at the base. But I can not agree that it’s a good idea.
I’m more going for a script on the new infa that interjects WS on the old …

But I’m just starting to dig :slight_smile:


#5

The Kaliop Migrations bundle seems very interesting.
But I do not see how he can handle the merging of databases.


#6

I haven’t used it, but my idea would be to first export content types from all but one database and import into the remaining one, and then export/import content in the same way.

@gggeek Is something like this possible?


#7

I will prefer to write my own export/import script when handling with many databases. I will then have a full control over my code and export files. Take care about relations, locations and embed contents IDs to avoid any conflict in the new DB.
If you will move to eZPlatform Enterprise , Support and Partner Teams in your region could help.


#8

eZ has a meta content database model (some people would call it EAV). Trying to use SQL directly with content will be very difficult and hackish. And you will not be sure whether you did it correctly. It makes more sense to use eZ Public PHP API directly or with above-mentioned migrations Bundle.


#9

Totally agree with you Gof. Out of the question to put your hands in the sql.

For the moment I have a WS on my old site which exports me the data in php format (var_export)

An eZPlatform side script that retrieves them.
But I have difficulties with the conversion of rich text.

To do the conversion I use (I hijack?) the command line ‘ezxmltext:convert-to-richtext’

    /** @var DatabaseHandler $dbHandler */ // @ezpublish.api.storage_engine.legacy.dbhandler
    $this->dbHandler = $container->get("ezpublish.api.storage_engine.legacy.dbhandler");
    /** @var Logger $logger */
    $this->logger = $container->get("logger"); // @?logger

    $this->converter = new ConvertXmlTextToRichTextCommand($this->dbHandler, $this->logger);
    $richText = $this->converter->convert($xmlText_from_old_site)

But the result is not up to my expectations.

[2018-03-19 10:07:37] app.ERROR: Validation errors when converting xmlstring
result

<?xml version="1.0" encoding="UTF-8"?>`
<section xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ezxhtml="http://ez.no/xmlns/ezpublish/docbook/xhtml" xmlns:ezcustom="http://ez.no/xmlns/ezpublish/docbook/custom" version="5.0-variant ezpublish-1.0">
<para>Près de 3 millions de vues en moins d'une semaine. C'est le succès incroyable remporté par la dernière vidéo du distributeur allemand Edeka. Un clip musical décalé faisant la promotion des MDD "très sexy" de l'enseigne.</para>
<para>Le distributeur s'est offert les services d'un artiste allemand un peu sur le retour, Friedrich Liechtenstein, pour susurrer d'un air langoureux un refrain que l'on peut traduire par "super mignon, super sexy, super facile, super lubrique" (le dernier mot signifiant selon les cas "excitant" ou "excité")...</para>
<para>Le phénomène fait le tour du web cette semaine et accumule les commentaires enthousiastes. Une vraie réussite virale.</para>
<para> </para> 
&lt;iframe width="640" height="360" src="//www.youtube-nocookie.com/embed/jxVcgDMBU94?rel=0" frameborder="0" allowfullscreen&gt;&lt;/iframe&gt;
<para> </para>
</section>\n

errors
Error in 0:0: Element section has extra content: para

xmlString

<?xml version="1.0" encoding="utf-8"?>
<section xmlns:image="http://ez.no/namespaces/ezpublish3/image/" xmlns:xhtml="http://ez.no/namespaces/ezpublish3/xhtml/" xmlns:custom="http://ez.no/namespaces/ezpublish3/custom/">
<paragraph>Près de 3 millions de vues en moins d'une semaine. C'est le succès incroyable remporté par la dernière vidéo du distributeur allemand Edeka. Un clip musical décalé faisant la promotion des MDD "très sexy" de l'enseigne.</paragraph>
<paragraph>Le distributeur s'est offert les services d'un artiste allemand un peu sur le retour, Friedrich Liechtenstein, pour susurrer d'un air langoureux un refrain que l'on peut traduire par "super mignon, super sexy, super facile, super lubrique" (le dernier mot signifiant selon les cas "excitant" ou "excité")...</paragraph>
<paragraph>Le phénomène fait le tour du web cette semaine et accumule les commentaires enthousiastes. Une vraie réussite virale.</paragraph>
<paragraph> </paragraph>
<paragraph xmlns:tmp="http://ez.no/namespaces/ezpublish3/temporary/">
  <literal class="html"> &lt;iframe width="640" height="360" src="//www.youtube-nocookie.com/embed/jxVcgDMBU94?rel=0" frameborder="0" allowfullscreen&gt;&lt;/iframe&gt;</literal>
</paragraph>
<paragraph> </paragraph>
</section>\n

This results in errors when validating the data for registration.

class eZ\Publish\Core\FieldType\ValidationError
     string(87) "Validation of XML content failed:
            Error in 2:0: Element section has extra content: para"

#10

Huh, not sure if I can help there. Sound like either some bad ezxml or bug in the converter :confused:


#11

As far as i had to work with ricthText, i had to contains every text into para. Wich means “Près de […] \n” in your case.
An other step is to remove all extra character such as “\n”

In addition to that, basically the xml validator is really hard to understand and, in my case, i removed every html tags or special characters, which meant a lot of content loss, for it to agree my content…

Regards,


#12

My code : https://github.com/RemySMILE/ImportFromMultiEZ4toPlatformBundle


#13

I mean of course using the API for import/export (ContentService, ContentTypeService, …), no sql query!


#14

I updated my code.

I change my strategy a little bit.
I will start by recovering all the arbo donations I need without dealing with the rich texts.

I will treat the conversion in a second time.


#15

Exactly. There are the param elements. And the error message tells me he does not know what to do with it.

<para>Près de 3 millions ... </para>


#16

I just updated my message. We can see more clearly.
And indeed, there is a pb with the iframe …


#17

The way we have done something similar was the following:
a) write custom code to export all content in the yml format compatible with ezmigrationbundle
b) use migrationbudle for import

The advantage that this brings is that the yml makes it easy to troubleshoot/debug what is going on, more than direct calls to rest-api would. Also it makes it easy to transfer a lot of data, as often if you run 300K http calls in a row a few do fail because of net or other stuff.

The complex parts were:
a) as any complex migration involves many passes, because of bugs and because of new content added to the origin system, we had to implement a way to do ‘upserts’ in migration bundle, ie. if content X was already imported in one pass, update it in another pass instead of trying to recreate it
b) handling of object relations. This was a bit more tricky: for every import execute 2 phases: on first phase create contents but transform all obj relations into relations to a placeholder-object (this allows to create all contents), then do in the 2nd phase only updates that fix all obj-relations (incl. of course rich-text embeds)
c) for our case, the content classes definitions between source and target were different, so for the ‘export’ phase we had to write code that transformed the existing contents into the new classes. This was not hard per se, but it turned out that the original content did not respect the content-class definitions (because the site had a long history, and as you know ez does not revalidate contents if they are not edited after their class def gets changed), and that the transformation rules were specified in an incomplete and buggy way because no-one knew in depth all the details of the hundreds of thousands of existing objects. So e spent a lot of time polishing and rewriting the export code. In a couple of cases, of course, even the target classes definitions did change during the migration process, because of business requirements for the new site…

The public version of the migration bundle was evolved to implement some of the stuff that was useful for this, but not all of it, such as:
a) it can now by itself easily export yml representation of existing content
b) it can now scan for migrations to import multiple ymls from a nested directory structure (good if you want to import more than a dozen thousand objects)

If doing things this way seems a good idea, I’d be happy to offer some support for implementing it.
I might even publish a ‘contentMigrationSupportBundle’ at some point in time, but I am quite busy at the moment, and tbh I’d rather do a paid-for consulting exercise rather than just throw to the community some code that needs a great deal of docs and discussion to be used in a fruitful way…