Mydumper & Myloader Documentation

Mydumper.org web site has been missing in action for a while now. I've uploaded a copy of the Mydumper documentation to this web site for easy reference. I did benchmarks on my blog athttp://vbtechsupport.com/1695/ and http://vbtechsupport.com/1716/

Note: Centmin Mod only hosting documentation and provided as is without any technical support.

Download:

Download latest Mydumper version from https://launchpad.net/mydumper. You may also need aMydumper patch for Mydumper to work with MariaDB 5.5 MySQL.

• authors
• compiling
• mydumper usage
• myloader usage
• output files
• examples

Authors

The code for mydumper has been written by the following people:

• Domas Mituzas <http://dom.as/> , Facebook ( domas at fb dot com )
• Andrew Hutchings <http://www.linuxjedi.co.uk> , SkySQL ( andrew at skysql dot com )
• Mark Leith <http://www.markleith.co.uk/> , Oracle Corporation ( mark dot leith at oracle dot com )
  

  Compiling

  Requirements
  ------------

  mydumper requires the following before it can be compiled:

• CMake <http://www.cmake.org/>
• Glib2 <http://www.gtk.org/index.php> (with development packages)
• PCRE <http://www.pcre.org/> (with development packages)
• MySQL <http://www.mysql.com/> client libraries (with development packages)

  Additionally the following packages are optional:

• python-sphinx <http://sphinx.pocoo.org/> (for documentation)

  Ubuntu/Debian
  ^^^^^^^^^^^^^

     apt-get install libglib2.0-dev libmysqlclient15-dev zlib1g-dev libpcre3-dev
  

  Fedora/Redhat/CentOS
  ^^^^^^^^^^

     yum install glib2-devel mysql-devel zlib-devel pcre-devel
  

  OpenSUSE
  ^^^^

     zypper install glib2-devel libmysqlclient-devel pcre-devel zlib-devel
  

  Mac OSX
  ^^^^^^^

     port install glib2 mysql5 pcre
  

  CMake
  -----

  CMake is used for mydumper's build system and is executed as follows::

    cmake .
    make
  

  You can optionally provide parameters for CMake, the possible options are:

• -DMYSQL_CONFIG=/path/to/mysql_config - The path and filename for the mysql_config executable
• -DCMAKE_INSTALL_PREFIX=/install/path - The path where mydumper should be installed

  Documentation
  -------------

  If you wish to just compile the documentation you can do so with::

    cmake .
    make doc_html
  

  or for a man page output::

    cmake .
    make doc_man
  

  ---

  Mydumper Usage

  Synopsis
  --------

  mydumper [OPTION...]

  Description
  -----------

  mydumper is a tool used for backing up MySQL database servers much
  faster than the mysqldump tool distributed with MySQL. It also has the
  capability to retrieve the binary logs from the remote server at the same time
  as the dump itself. The advantages of mydumper are:

• Parallelism (hence, speed) and performance (avoids expensive character set conversion routines, efficient code overall)
• Easier to manage output (separate files for tables, dump metadata, etc, easy to view/parse data)
• Consistency - maintains snapshot across all threads, provides accurate master and slave log positions, etc
• Manageability - supports PCRE for specifying database and tables inclusions and exclusions

  Options
  -------

  The mydumper tool has several available options:

  mydumper

   --help, -?

  Show help text

   --host, -h

  Hostname of MySQL server to connect to (default localhost)

   --user, -u

  MySQL username with the correct privileges to execute the dump

   --password, -p

  The corresponding password for the MySQL user

   --port, -P

  The port for the MySQL connection.

  Note:For localhost TCP connections use 127.0.0.1 for --host`.

   --socket, -S

  The UNIX domain socket file to use for the connection

   --database, -B

  Database to dump

   --table-list, -T

  A comma separated list of tables to dump

   --threads, -t

  The number of threads to use for dumping data, default is 4

  Note:Other threads are used in mydumper, this option does not control these

   --outputdir, -o

  Output directory name, default is export-YYYYMMDD-HHMMSS

   --statement-size, -s

  The maximum size for an insert statement before breaking into a new
  statement, default 1,000,000 bytes

   --rows, -r

  Split table into chunks of this many rows, default unlimited

   --compress, -c

  Compress the output files

   --compress-input, -C

  Use client protocol compression for connections to the MySQL server

   --build-empty-files, -e

  Create empty dump files if there is no data to dump

   --regex, -x

  A regular expression to match against database and table

   --ignore-engines, -i

  Comma separated list of storage engines to ignore

   --no-schemas, -m

  Do not dump schemas with the data

   --long-query-guard, -l

  Timeout for long query execution in seconds, default 60

   --kill-long-queries, -k

  Kill long running queries instead of aborting the dump

   --version, -V

  Show the program version and exit

   --verbose, -v

  The verbosity of messages. 0 = silent, 1 = errors, 2 = warnings, 3 = info.
  Default is 2.

   --binlogs, -b

  Get the binlogs from the server as well as the dump files

    --daemon, -D

  Enable daemon mode

   --snapshot-interval, -I

  Interval between each dump snapshot (in minutes), requires
  --daemon`, default 60 (minutes)

   --logfile, -L

  A file to log mydumper output to instead of console output. Useful for
  daemon mode.

   --no-locks, -k

  Do not execute the temporary shared read lock.

  .. warning::

  This will cause inconsistent backups.

  ---

  Myloader Usage

  Synopsis
  --------

  myloader [OPTION...]

  Description
  -----------

  myloader is a tool used for multi-threaded restoration of mydumper
  backups.

  Options
  -------

  The myloader tool has several available options:

  myloader

   --help, -?

  Show help text

   --host, -h

  Hostname of MySQL server to connect to (default localhost)

   --user, -u

  MySQL username with the correct privileges to execute the restoration

   --password, -p

  The corresponding password for the MySQL user

   --port, -P

  The port for the MySQL connection.

  Note:For localhost TCP connections use 127.0.0.1 for --host`.

   --socket, -S

  The UNIX domain socket file to use for the connection

   --threads, -t

  The number of threads to use for restoring data, default is 4

   --version, -V

  Show the program version and exit

   --compress-protocol, -C

  Use client protocol compression for connections to the MySQL server

   --directory, -d

  The directory of the mydumper backup to restore

   --database, -B

  An alternative database to load the dump into

  Note:For use with single database dumps. When using with multi-database dumps
  that have duplicate table names in more than one database it may cause 
  errors. Alternatively this scenario may give unpredictable results with
  --overwrite-tables`.

   --queries-per-transaction, -q

  Number of INSERT queries to execute per transaction during restore, default
  is 1000.

   --overwrite-tables, -o

  Drop any existing tables when restoring schemas

   --enable-binlog, -e

  Log the data loading in the MySQL binary log if enabled (off by default)

   --verbose, -v

  The verbosity of messages. 0 = silent, 1 = errors, 2 = warnings, 3 = info.
  Default is 2.

  ---

  Output Files

  mydumper generates several files during the generation of the dump. Many of
  these are for the table data itself since every table has at least one file.

  Metadata
  --------
  When a dump is executed a file called ``.metadata`` is created in the output
  directory. This contains the start and end time of the dump as well as the
  master binary log positions if applicable.

  This is an example of the content of this file::

    Started dump at: 2011-05-05 13:57:17
    SHOW MASTER STATUS:
    Log: linuxjedi-laptop-bin.000001
    Pos: 106
  
    Finished dump at: 2011-05-05 13:57:17
  

  Table Data
  ----------
  The data from every table is written into a separate file, also if the
  --rows <mydumper --rows>` option is used then each chunk of table will
  be in a separate file. The file names for this are in the format::

      database.table.sql(.gz)
  

  or if chunked::

      database.table.chunk.sql(.gz)
  

  Where 'chunk' is a number padded with up to 5 zeros.

  Table Schemas
  -------------
  When the --schemas <mydumper --schemas>` option is used mydumper will
  create a file for the schema of every table it is writing data for. The files
  for this are in the following format::

      database.table-schema.sql(.gz)
  

  Binary Logs
  -----------
  Binary logs are retrieved when --binlogs <mydumper --binlogs>` option
  has been set. This will store them in the ``binlog_snapshot/`` sub-directory
  inside the dump directory.

  The binary log files have the same filename as the MySQL server that supplies them and will also have a .gz on the end if they are compressed.

  Daemon mode
  -----------
  Daemon mode does things a little differently. There are the directories ``0``
  and ``1`` inside the dump directory. These alternate when dumping so that if
  mydumper fails for any reason there is still a good snapshot. When a snapshot
  dump is complete the ``last_dump`` symlink is updated to point to that dump.

  If binary logging is enabled mydumper will connect as if it is a slave server
  and constantly retreives the binary logs into the ``binlogs`` subdirectory.

  ---

  Examples

  Simple Usage
  ------------
  Just running mydumper without any options will try to connect to a
  server using the default socket path. It will then dump the tables from all
  databases using 4 worker threads.

  Regex
  -----
  To use mydumper's regex feature simply use the
  --regex <mydumper --regex>` option. In the following example mydumper
  will ignore the ``test`` and ``mysql`` databases::

      mydumper --regex '^(?!(mysql|test))'
  

  Restoring a dump
  ----------------
  Mydumper now include myloader which is a multi-threaded restoration tool. To
  use myloader with a mydumper dump you simply need to pass it the directory of
  the dump along with a user capable of restoring the schemas and data. As an
  example the following will restore a dump overwriting any existing tables::

      myloader --directory=export-20110614-094953 --overwrite-tables --user=root
  

  Daemon mode
  -----------
  Mydumper has a daemon mode which will snapshot the dump data every so often
  whilst continuously retreiving the binary log files. This gives a continuous
  consistent backup right up to the point where the database server fails. To use
  this you simply need to use the --daemon <mydumper --daemon>` option.

  In the following example mydumper will use daemon mode, creating a snapshot
  every half an hour and log to an output file::

      mydumper --daemon --snapshot-interval=30 --logfile=dump.log
  

  ---

mydumper用例

# mydumper -B ttlsa_com -o /home/data/bak -b
# ls
binlog_snapshot metadata ttlsa_com.my_area-schema.sql ttlsa_com.my_area.sql
# cat metadata
Started dump at: 2012-02-20 11:40:21
SHOW MASTER STATUS:
Log: mysqld-bin.000001
Pos: 191363
Finished dump at: 2012-02-20 11:40:22
# mydumper -B ttlsa_com -o /home/data/bak -b -D --snapshot-interval=30
# myloader -d /home/data/bak/last_dump -o -B ttlsa_com //还原

参考：

http://centminmod.com/mydumper.html

http://www.ttlsa.com/mysql/mysql-backup-tool-mydumper/