“Resource deadlock avoided during COMMIT” when a node leaves the group












0















I believe it's a bug in MariaDB Galera (10.3) but wanted to confirm. I have 3 nodes in the cluster (A, B, C) and a sysbench running the OLTP write-only test connecting only to node "A". When the node "B" is being shutdown gracefully with systemctl stop mariadb the sysbench is always immediately interrupted, showing error lines below (varying number depending on the run).



I expected sysbench to be completely unaware of node "A" leaving the group.



Sysbench reported error:



Initializing worker threads...

Threads started!

FATAL: mysql_stmt_execute() returned error 1180 (Got error 35 "Resource deadlock avoided" during COMMIT) for query 'COMMIT'

FATAL: `thread_run' function failed: /usr/share/sysbench/oltp_common.lua:405: SQL error, errno = 1180, state = 'HY000': Got error 35 "Resource deadlock avoided" during COMMIT

FATAL: mysql_stmt_execute() returned error 1180 (Got error 35 "Resource deadlock avoided" during COMMIT) for query 'COMMIT'

FATAL: `thread_run' function failed: /usr/share/sysbench/oltp_common.lua:405: SQL error, errno = 1180, state = 'HY000': Got error 35 "Resource deadlock avoided" during COMMIT

FATAL: mysql_stmt_execute() returned error 1180 (Got error 35 "Resource deadlock avoided" during COMMIT) for query 'COMMIT'

FATAL: `thread_run' function failed: /usr/share/sysbench/oltp_common.lua:405: SQL error, errno = 1180, state = 'HY000': Got error 35 "Resource deadlock avoided" during COMMIT


sysbench invocation:



sysbench --mysql-user='sbench' 

--mysql-password='test' 

--db-driver=mysql 

--mysql-host=$TEST_HOST_A 

--mysql-db=$TEST_DB_NAME 

--time=300 

--threads=16 

oltp_write_only 

run


When this happens one of the other 2-nodes reports:



Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: declaring eb2c2526 at tcp://10.4.153.11:4567 stable

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: forgetting f3789450 (tcp://10.4.151.8:4567)

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Node 33129c3a state prim

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: view(view_id(PRIM,33129c3a,32) memb {

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         33129c3a,0

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         eb2c2526,0

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: } joined {

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: } left {

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: } partitioned {

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         f3789450,0

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: })

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: save pc into disk

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: forgetting f3789450 (tcp://10.4.151.8:4567)

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: fc356030-3326-11e9-b854-fa77c4586cf9

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE EXCHANGE: sent state msg: fc356030-3326-11e9-b854-fa77c4586cf9

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE EXCHANGE: got state msg: fc356030-3326-11e9-b854-fa77c4586cf9 from 0 (ip-10-4-20-211)

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE EXCHANGE: got state msg: fc356030-3326-11e9-b854-fa77c4586cf9 from 1 (ip-10-4-153-11)

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Quorum results:

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         version    = 4,

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         component  = PRIMARY,

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         conf_id    = 29,

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         members    = 2/2 (joined/total),

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         act_id     = 17299676,

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         last_appl. = 17299555,

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         protocols  = 0/9/3 (gcs/repl/appl),

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         group UUID = 3c6c83e4-3286-11e9-945b-e253cb1ea9e0

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Flow-control interval: [23, 23]

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Trying to continue unpaused monitor

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: New cluster view: global state: 3c6c83e4-3286-11e9-945b-e253cb1ea9e0:17299676, view# 30: Primary, number of nodes: 2, my index: 0, protocol version 3

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: REPL Protocols: 9 (4, 2)

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: Assign initial position for certification: 17299676, protocol version: 4

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Service thread queue flushed.

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 122 [Warning] Aborted connection 122 to db: 'sbtest' user: 'sbench' host: 'ip-10-4-20-121.us-east-2.compute.internal' (Got an error reading communication packets)

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 115 [Warning] Aborted connection 115 to db: 'sbtest' user: 'sbench' host: 'ip-10-4-20-121.us-east-2.compute.internal' (Got an error reading communication packets)

Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 124 [Warning] Aborted connection 124 to db: 'sbtest' user: 'sbench' host: 'ip-10-4-20-121.us-east-2.compute.internal' (Got an error reading communication packets)

Feb 18 02:43:39 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:39 0 [Note] WSREP:  cleaning up f3789450 (tcp://10.4.151.8:4567)


Other details:




  • OS: Ubuntu 18.04.2 LTS

  • MariaDB: 10.3.12-MariaDB-1:10.3.12+maria~bionic-log mariadb.org binary distribution






galera






share



























    0















    I believe it's a bug in MariaDB Galera (10.3) but wanted to confirm. I have 3 nodes in the cluster (A, B, C) and a sysbench running the OLTP write-only test connecting only to node "A". When the node "B" is being shutdown gracefully with systemctl stop mariadb the sysbench is always immediately interrupted, showing error lines below (varying number depending on the run).



    I expected sysbench to be completely unaware of node "A" leaving the group.



    Sysbench reported error:



    Initializing worker threads...
    
Threads started!
    
FATAL: mysql_stmt_execute() returned error 1180 (Got error 35 "Resource deadlock avoided" during COMMIT) for query 'COMMIT'
    
FATAL: `thread_run' function failed: /usr/share/sysbench/oltp_common.lua:405: SQL error, errno = 1180, state = 'HY000': Got error 35 "Resource deadlock avoided" during COMMIT
    
FATAL: mysql_stmt_execute() returned error 1180 (Got error 35 "Resource deadlock avoided" during COMMIT) for query 'COMMIT'
    
FATAL: `thread_run' function failed: /usr/share/sysbench/oltp_common.lua:405: SQL error, errno = 1180, state = 'HY000': Got error 35 "Resource deadlock avoided" during COMMIT
    
FATAL: mysql_stmt_execute() returned error 1180 (Got error 35 "Resource deadlock avoided" during COMMIT) for query 'COMMIT'
    
FATAL: `thread_run' function failed: /usr/share/sysbench/oltp_common.lua:405: SQL error, errno = 1180, state = 'HY000': Got error 35 "Resource deadlock avoided" during COMMIT


    sysbench invocation:



    sysbench --mysql-user='sbench' 
    
--mysql-password='test' 
    
--db-driver=mysql 
    
--mysql-host=$TEST_HOST_A 
    
--mysql-db=$TEST_DB_NAME 
    
--time=300 
    
--threads=16 
    
oltp_write_only 
    
run


    When this happens one of the other 2-nodes reports:



    Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: declaring eb2c2526 at tcp://10.4.153.11:4567 stable
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: forgetting f3789450 (tcp://10.4.151.8:4567)
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Node 33129c3a state prim
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: view(view_id(PRIM,33129c3a,32) memb {
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         33129c3a,0
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         eb2c2526,0
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: } joined {
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: } left {
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: } partitioned {
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         f3789450,0
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: })
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: save pc into disk
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: forgetting f3789450 (tcp://10.4.151.8:4567)
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: fc356030-3326-11e9-b854-fa77c4586cf9
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE EXCHANGE: sent state msg: fc356030-3326-11e9-b854-fa77c4586cf9
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE EXCHANGE: got state msg: fc356030-3326-11e9-b854-fa77c4586cf9 from 0 (ip-10-4-20-211)
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE EXCHANGE: got state msg: fc356030-3326-11e9-b854-fa77c4586cf9 from 1 (ip-10-4-153-11)
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Quorum results:
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         version    = 4,
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         component  = PRIMARY,
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         conf_id    = 29,
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         members    = 2/2 (joined/total),
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         act_id     = 17299676,
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         last_appl. = 17299555,
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         protocols  = 0/9/3 (gcs/repl/appl),
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         group UUID = 3c6c83e4-3286-11e9-945b-e253cb1ea9e0
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Flow-control interval: [23, 23]
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Trying to continue unpaused monitor
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: New cluster view: global state: 3c6c83e4-3286-11e9-945b-e253cb1ea9e0:17299676, view# 30: Primary, number of nodes: 2, my index: 0, protocol version 3
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: REPL Protocols: 9 (4, 2)
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: Assign initial position for certification: 17299676, protocol version: 4
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Service thread queue flushed.
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 122 [Warning] Aborted connection 122 to db: 'sbtest' user: 'sbench' host: 'ip-10-4-20-121.us-east-2.compute.internal' (Got an error reading communication packets)
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 115 [Warning] Aborted connection 115 to db: 'sbtest' user: 'sbench' host: 'ip-10-4-20-121.us-east-2.compute.internal' (Got an error reading communication packets)
    
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 124 [Warning] Aborted connection 124 to db: 'sbtest' user: 'sbench' host: 'ip-10-4-20-121.us-east-2.compute.internal' (Got an error reading communication packets)
    
Feb 18 02:43:39 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:39 0 [Note] WSREP:  cleaning up f3789450 (tcp://10.4.151.8:4567)


    Other details:




    • OS: Ubuntu 18.04.2 LTS

    • MariaDB: 10.3.12-MariaDB-1:10.3.12+maria~bionic-log mariadb.org binary distribution






    galera






    share

























      0












      0








      0








      I believe it's a bug in MariaDB Galera (10.3) but wanted to confirm. I have 3 nodes in the cluster (A, B, C) and a sysbench running the OLTP write-only test connecting only to node "A". When the node "B" is being shutdown gracefully with systemctl stop mariadb the sysbench is always immediately interrupted, showing error lines below (varying number depending on the run).



      I expected sysbench to be completely unaware of node "A" leaving the group.



      Sysbench reported error:



      Initializing worker threads...
      
Threads started!
      
FATAL: mysql_stmt_execute() returned error 1180 (Got error 35 "Resource deadlock avoided" during COMMIT) for query 'COMMIT'
      
FATAL: `thread_run' function failed: /usr/share/sysbench/oltp_common.lua:405: SQL error, errno = 1180, state = 'HY000': Got error 35 "Resource deadlock avoided" during COMMIT
      
FATAL: mysql_stmt_execute() returned error 1180 (Got error 35 "Resource deadlock avoided" during COMMIT) for query 'COMMIT'
      
FATAL: `thread_run' function failed: /usr/share/sysbench/oltp_common.lua:405: SQL error, errno = 1180, state = 'HY000': Got error 35 "Resource deadlock avoided" during COMMIT
      
FATAL: mysql_stmt_execute() returned error 1180 (Got error 35 "Resource deadlock avoided" during COMMIT) for query 'COMMIT'
      
FATAL: `thread_run' function failed: /usr/share/sysbench/oltp_common.lua:405: SQL error, errno = 1180, state = 'HY000': Got error 35 "Resource deadlock avoided" during COMMIT


      sysbench invocation:



      sysbench --mysql-user='sbench' 
      
--mysql-password='test' 
      
--db-driver=mysql 
      
--mysql-host=$TEST_HOST_A 
      
--mysql-db=$TEST_DB_NAME 
      
--time=300 
      
--threads=16 
      
oltp_write_only 
      
run


      When this happens one of the other 2-nodes reports:



      Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: declaring eb2c2526 at tcp://10.4.153.11:4567 stable
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: forgetting f3789450 (tcp://10.4.151.8:4567)
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Node 33129c3a state prim
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: view(view_id(PRIM,33129c3a,32) memb {
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         33129c3a,0
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         eb2c2526,0
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: } joined {
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: } left {
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: } partitioned {
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         f3789450,0
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: })
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: save pc into disk
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: forgetting f3789450 (tcp://10.4.151.8:4567)
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: fc356030-3326-11e9-b854-fa77c4586cf9
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE EXCHANGE: sent state msg: fc356030-3326-11e9-b854-fa77c4586cf9
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE EXCHANGE: got state msg: fc356030-3326-11e9-b854-fa77c4586cf9 from 0 (ip-10-4-20-211)
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE EXCHANGE: got state msg: fc356030-3326-11e9-b854-fa77c4586cf9 from 1 (ip-10-4-153-11)
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Quorum results:
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         version    = 4,
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         component  = PRIMARY,
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         conf_id    = 29,
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         members    = 2/2 (joined/total),
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         act_id     = 17299676,
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         last_appl. = 17299555,
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         protocols  = 0/9/3 (gcs/repl/appl),
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         group UUID = 3c6c83e4-3286-11e9-945b-e253cb1ea9e0
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Flow-control interval: [23, 23]
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Trying to continue unpaused monitor
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: New cluster view: global state: 3c6c83e4-3286-11e9-945b-e253cb1ea9e0:17299676, view# 30: Primary, number of nodes: 2, my index: 0, protocol version 3
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: REPL Protocols: 9 (4, 2)
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: Assign initial position for certification: 17299676, protocol version: 4
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Service thread queue flushed.
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 122 [Warning] Aborted connection 122 to db: 'sbtest' user: 'sbench' host: 'ip-10-4-20-121.us-east-2.compute.internal' (Got an error reading communication packets)
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 115 [Warning] Aborted connection 115 to db: 'sbtest' user: 'sbench' host: 'ip-10-4-20-121.us-east-2.compute.internal' (Got an error reading communication packets)
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 124 [Warning] Aborted connection 124 to db: 'sbtest' user: 'sbench' host: 'ip-10-4-20-121.us-east-2.compute.internal' (Got an error reading communication packets)
      
Feb 18 02:43:39 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:39 0 [Note] WSREP:  cleaning up f3789450 (tcp://10.4.151.8:4567)


      Other details:




      • OS: Ubuntu 18.04.2 LTS

      • MariaDB: 10.3.12-MariaDB-1:10.3.12+maria~bionic-log mariadb.org binary distribution






      galera






      share














      I believe it's a bug in MariaDB Galera (10.3) but wanted to confirm. I have 3 nodes in the cluster (A, B, C) and a sysbench running the OLTP write-only test connecting only to node "A". When the node "B" is being shutdown gracefully with systemctl stop mariadb the sysbench is always immediately interrupted, showing error lines below (varying number depending on the run).



      I expected sysbench to be completely unaware of node "A" leaving the group.



      Sysbench reported error:



      Initializing worker threads...
      
Threads started!
      
FATAL: mysql_stmt_execute() returned error 1180 (Got error 35 "Resource deadlock avoided" during COMMIT) for query 'COMMIT'
      
FATAL: `thread_run' function failed: /usr/share/sysbench/oltp_common.lua:405: SQL error, errno = 1180, state = 'HY000': Got error 35 "Resource deadlock avoided" during COMMIT
      
FATAL: mysql_stmt_execute() returned error 1180 (Got error 35 "Resource deadlock avoided" during COMMIT) for query 'COMMIT'
      
FATAL: `thread_run' function failed: /usr/share/sysbench/oltp_common.lua:405: SQL error, errno = 1180, state = 'HY000': Got error 35 "Resource deadlock avoided" during COMMIT
      
FATAL: mysql_stmt_execute() returned error 1180 (Got error 35 "Resource deadlock avoided" during COMMIT) for query 'COMMIT'
      
FATAL: `thread_run' function failed: /usr/share/sysbench/oltp_common.lua:405: SQL error, errno = 1180, state = 'HY000': Got error 35 "Resource deadlock avoided" during COMMIT


      sysbench invocation:



      sysbench --mysql-user='sbench' 
      
--mysql-password='test' 
      
--db-driver=mysql 
      
--mysql-host=$TEST_HOST_A 
      
--mysql-db=$TEST_DB_NAME 
      
--time=300 
      
--threads=16 
      
oltp_write_only 
      
run


      When this happens one of the other 2-nodes reports:



      Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: declaring eb2c2526 at tcp://10.4.153.11:4567 stable
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: forgetting f3789450 (tcp://10.4.151.8:4567)
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Node 33129c3a state prim
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: view(view_id(PRIM,33129c3a,32) memb {
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         33129c3a,0
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         eb2c2526,0
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: } joined {
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: } left {
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: } partitioned {
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         f3789450,0
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: })
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: save pc into disk
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: forgetting f3789450 (tcp://10.4.151.8:4567)
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: fc356030-3326-11e9-b854-fa77c4586cf9
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE EXCHANGE: sent state msg: fc356030-3326-11e9-b854-fa77c4586cf9
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE EXCHANGE: got state msg: fc356030-3326-11e9-b854-fa77c4586cf9 from 0 (ip-10-4-20-211)
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: STATE EXCHANGE: got state msg: fc356030-3326-11e9-b854-fa77c4586cf9 from 1 (ip-10-4-153-11)
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Quorum results:
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         version    = 4,
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         component  = PRIMARY,
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         conf_id    = 29,
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         members    = 2/2 (joined/total),
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         act_id     = 17299676,
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         last_appl. = 17299555,
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         protocols  = 0/9/3 (gcs/repl/appl),
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]:         group UUID = 3c6c83e4-3286-11e9-945b-e253cb1ea9e0
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Flow-control interval: [23, 23]
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Trying to continue unpaused monitor
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: New cluster view: global state: 3c6c83e4-3286-11e9-945b-e253cb1ea9e0:17299676, view# 30: Primary, number of nodes: 2, my index: 0, protocol version 3
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: REPL Protocols: 9 (4, 2)
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 23 [Note] WSREP: Assign initial position for certification: 17299676, protocol version: 4
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 0 [Note] WSREP: Service thread queue flushed.
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 122 [Warning] Aborted connection 122 to db: 'sbtest' user: 'sbench' host: 'ip-10-4-20-121.us-east-2.compute.internal' (Got an error reading communication packets)
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 115 [Warning] Aborted connection 115 to db: 'sbtest' user: 'sbench' host: 'ip-10-4-20-121.us-east-2.compute.internal' (Got an error reading communication packets)
      
Feb 18 02:43:34 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:34 124 [Warning] Aborted connection 124 to db: 'sbtest' user: 'sbench' host: 'ip-10-4-20-121.us-east-2.compute.internal' (Got an error reading communication packets)
      
Feb 18 02:43:39 ip-10-4-20-211 mysqld[10268]: 2019-02-18  2:43:39 0 [Note] WSREP:  cleaning up f3789450 (tcp://10.4.151.8:4567)


      Other details:




      • OS: Ubuntu 18.04.2 LTS

      • MariaDB: 10.3.12-MariaDB-1:10.3.12+maria~bionic-log mariadb.org binary distribution






      galera




      galera





      share












      share










      share



      share










      asked 50 secs ago









      SlawomirSlawomir

      20119




      20119






















          0






          active

          oldest

          votes











          Your Answer








          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "182"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: false,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: null,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f230007%2fresource-deadlock-avoided-during-commit-when-a-node-leaves-the-group%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          0






          active

          oldest

          votes








          0






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes
















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Database Administrators Stack Exchange!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f230007%2fresource-deadlock-avoided-during-commit-when-a-node-leaves-the-group%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          ف. موراي أبراهام

          Image
          ف. موراي أبراهام F. Murray Abraham ف. موراي أبراهام في 2008 معلومات شخصية الاسم عند الولادة فريد مراد إبراهيم Fahrid Murray Abraham الميلاد 24 أكتوبر 1939 (العمر 79 سنة) شاكبيتسبرغ، بنسيلفانيا مواطنة الولايات المتحدة  أسماء أخرى فرانك موراي أبراهام Frank Murray Abraham الحياة العملية المدرسة الأم جامعة تكساس في أوستن  التلامذة المشهورين جون كيل [1]   المهنة ممثل تلفزيوني،  وممثل أفلام،  وممثل تعبيري،  وممثل مسرحي  سنوات النشاط 1971–الآن موظف في كلية بروكلين  الجوائز جائزة الأوسكار لأفضل ممثل  (عن عمل:أماديوس)   المواقع IMDB صفحته على IMDB  السينما.كوم صفحته على السينما.كوم  تعديل   ف. موراي أبراهام (بالإنجليزية: F. Murray Abraham ) ، ممثل أمريكي من اصل سوري أرثوذكسي من ناحية الاب وإيطالي من ناحية الام وهو ممثل أمريكي من مواليد 24 أكتوبر 1939، حاصل على جائزة الأوسكار 1984 كأفضل ممثل عن دوره في فيلم أماديوس كما حصل بنفس الدور على جائزة الغولدن غلوب 1985 كأفضل ممثل في فيلم د...
          Read more

          صرب

          Image
          صرب التعداد الكلي 11.5–12.5 مليون [a] مناطق الوجود المميزة يبلغ عدد سكانها أكثر 100,000 في يوغوسلافيا السابقة   صربيا (excl. كوسوفو) 5,988,150 (2011)   البوسنة والهرسك 1,086,733 (2013) [1]   الجبل الأسود b 178,110 (2011) [2]   كوسوفو 90–120,000 (2012 قالب:Estimation) a   سلوفينيا 38,964 (2002) [3]   جمهورية مقدونيا 35,939 (2002) [4] بقية أوروبا · ·   ألمانيا بحدود 500,000 (2014 قالب:Estimation)   النمسا بحدود 300,000 (2010 قالب:Estimation) [5]   سويسرا بحدود 71,260 (2015 قالب:Estimation) [6]   فرنسا بحدود 120,000 (2002 قالب:Estimation) [7]   السويد بحدود 110–120,000 (قالب:Estimation)   المملكة المتحدة بحدود 70,000 (2001 قالب:Estimation) [8]   رومانيا 18,076 (2011) [9] أمريكا الشمالية ·   الولايات المتحدة c 199,080 (2012) [10]   كندا c 80,320 (2011) [11] بقية العالم ·   أستراليا c 69,544 (2011) [12] ...
          Read more

          كأس إنترتوتو

          Image
          كأس إنترتوتو شعار كأس إنترتوتو معلومات عامة الرياضة كرة القدم انطلقت 1968 (1995 من قبل اليويفا) انتهت 2008 المنظم الاتحاد الأوروبي لكرة القدم عدد الطبعات 35 نسخة رسمية تواترها سنوية عدد المشاركين 50 فريق الموقع الرسمي الموقع الرسمي الأكثر تتويجا هامبورغ شتوتغارت شالكه 04 نادي فياريال (لقبين) التسلسل الزمني للمنافسة تعديل   كأس إنترتوتو (بالإنجليزية: UEFA Intertoto Cup ) هي مسابقة كرة قدم صيفية لأندية كرة القدم الأوروبية التي لم تأهل للبطولات الكبرى المختلفة التي تقام سنوياً تحت رعاية الاتحاد الأوروبي لكرة القدم؛ مثل دوري أبطال أوروبا وكأس الاتحاد الأوروبي سابقاً. وقد أنشئت المسابقة في موسم 1961–62 كبطولة ودية تضم عدة أندية كرة قدم أوروبية، قبل أن يقوم الاتحاد الأوروبي لكرة القدم بإضافتها لقائمة البطولات المعترفة بها في عام 1995. وفي عام 2008 قام الاتحاد الأوروبي لكرة القدم بدمج بطولة كأس إنترتوتو مع بطولة كأس الاتحاد الأوروبي، [1] وبهذا أصبحت كل من بطولة كأس الاتحاد الأوروبي و كأس إنترتوتو وكأس الكؤوس الأوروبية مدموج...
          Read more