diff options
| author | Arthur Kepner <akepner@sgi.com> | 2007-03-11 07:35:15 +0100 |
|---|---|---|
| committer | Adrian Bunk <bunk@stusta.de> | 2007-03-11 07:35:15 +0100 |
| commit | 11bb9d392c1d5b63e2b9a0c8cc8a64cf49808757 (patch) | |
| tree | 65d9b830cb27c996b23b670a6c79b171f293bd98 /drivers/infiniband/hw/mthca/mthca_qp.c | |
| parent | ec333318b748e4f4ac3679a197620503b6d19d20 (diff) | |
| download | lwn-11bb9d392c1d5b63e2b9a0c8cc8a64cf49808757.tar.gz lwn-11bb9d392c1d5b63e2b9a0c8cc8a64cf49808757.zip | |
IB/mthca: Use mmiowb after doorbell ring
We discovered a problem when running IPoIB applications on multiple
CPUs on an Altix system. Many messages such as:
ib_mthca 0002:01:00.0: SQ 000014 full (19941644 head, 19941707 tail, 64 max, 0 nreq)
appear in syslog, and the driver wedges up.
Apparently this is because writes to the doorbells from different CPUs
reach the device out of order. The following patch adds mmiowb() calls
after doorbell rings to ensure the doorbell writes are ordered.
Signed-off-by: Arthur Kepner <akepner@sgi.com>
Signed-off-by: Roland Dreier <rolandd@cisco.com>
Signed-off-by: Adrian Bunk <bunk@stusta.de>
Diffstat (limited to 'drivers/infiniband/hw/mthca/mthca_qp.c')
| -rw-r--r-- | drivers/infiniband/hw/mthca/mthca_qp.c | 19 |
1 files changed, 19 insertions, 0 deletions
diff --git a/drivers/infiniband/hw/mthca/mthca_qp.c b/drivers/infiniband/hw/mthca/mthca_qp.c index fba608ed7df2..ac9ff2a35f62 100644 --- a/drivers/infiniband/hw/mthca/mthca_qp.c +++ b/drivers/infiniband/hw/mthca/mthca_qp.c @@ -39,6 +39,8 @@ #include <linux/string.h> #include <linux/slab.h> +#include <asm/io.h> + #include <rdma/ib_verbs.h> #include <rdma/ib_cache.h> #include <rdma/ib_pack.h> @@ -1724,6 +1726,11 @@ out: mthca_write64(doorbell, dev->kar + MTHCA_SEND_DOORBELL, MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock)); + /* + * Make sure doorbells don't leak out of SQ spinlock + * and reach the HCA out of order: + */ + mmiowb(); } qp->sq.next_ind = ind; @@ -1842,6 +1849,12 @@ out: qp->rq.next_ind = ind; qp->rq.head += nreq; + /* + * Make sure doorbells don't leak out of RQ spinlock and reach + * the HCA out of order: + */ + mmiowb(); + spin_unlock_irqrestore(&qp->rq.lock, flags); return err; } @@ -2099,6 +2112,12 @@ out: MTHCA_GET_DOORBELL_LOCK(&dev->doorbell_lock)); } + /* + * Make sure doorbells don't leak out of SQ spinlock and reach + * the HCA out of order: + */ + mmiowb(); + spin_unlock_irqrestore(&qp->sq.lock, flags); return err; } |