Age | Commit message (Collapse) | Author |
|
The spinlock and list head of fnic_list are initialized statically. It is
unnecessary to initialize them.
Link: https://lore.kernel.org/r/20210330125911.1050879-1-yangyingliang@huawei.com
Reported-by: Hulk Robot <hulkci@huawei.com>
Signed-off-by: Yang Yingliang <yangyingliang@huawei.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
Return a negative error code from the error handling case instead of 0 as
done elsewhere in this function.
Link: https://lore.kernel.org/r/1607068060-31203-1-git-send-email-zhangchangzhong@huawei.com
Fixes: 5df6d737dd4b ("[SCSI] fnic: Add new Cisco PCI-Express FCoE HBA")
Reported-by: Hulk Robot <hulkci@huawei.com>
Reviewed-by: Karan Tilak Kumar <kartilak@cisco.com>
Signed-off-by: Zhang Changzhong <zhangchangzhong@huawei.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
Avoid looping in fnic_scsi_abort_io() before sending fw reset when fnic is
in TRANS ETH state and when we have not received any link events.
Link: https://lore.kernel.org/r/20201121012145.18522-1-kartilak@cisco.com
Reviewed-by: Arulprabhu Ponnusamy <arulponn@cisco.com>
Co-developed-by: Satish Kharat <satishkh@cisco.com>
Signed-off-by: Satish Kharat <satishkh@cisco.com>
Signed-off-by: Karan Tilak Kumar <kartilak@cisco.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
This addresses the following coccinelle warning:
drivers/scsi/fnic/fnic_main.c:446:2-3: Unneeded semicolon
Link: https://lore.kernel.org/r/20200911091057.2938685-1-yanaijie@huawei.com
Reported-by: Hulk Robot <hulkci@huawei.com>
Signed-off-by: Jason Yan <yanaijie@huawei.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
This addresses the following gcc warning with "make W=1":
drivers/scsi/fnic/fnic_main.c: In function ‘fnic_set_vlan’:
drivers/scsi/fnic/fnic_main.c:555:6: warning: variable ‘old_vlan’ set
but not used [-Wunused-but-set-variable]
555 | u16 old_vlan;
| ^~~~~~~~
Link: https://lore.kernel.org/r/20200831081126.3251288-2-yanaijie@huawei.com
Reported-by: Hulk Robot <hulkci@huawei.com>
Signed-off-by: Jason Yan <yanaijie@huawei.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
Fix the following sparse warning:
drivers/scsi/fnic/fnic_main.c:52:1: warning: symbol 'fnic_list' was not
declared. Should it be static?
drivers/scsi/fnic/fnic_main.c:53:1: warning: symbol 'fnic_list_lock' was
not declared. Should it be static?
Link: https://lore.kernel.org/r/20200415093809.9365-2-yanaijie@huawei.com
Reported-by: Hulk Robot <hulkci@huawei.com>
Signed-off-by: Jason Yan <yanaijie@huawei.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
When calling debugfs functions, there is no need to ever check the return
value. The function can work or not, but the code logic should never do
something different based on this.
Cc: Satish Kharat <satishkh@cisco.com>
Cc: Sesidhar Baddela <sebaddel@cisco.com>
Cc: Karan Tilak Kumar <kartilak@cisco.com>
Cc: "James E.J. Bottomley" <jejb@linux.ibm.com>
Cc: "Martin K. Petersen" <martin.petersen@oracle.com>
Cc: linux-scsi@vger.kernel.org
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Acked-by: Satish Kharat <satishkh@cisco.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
This patch adds changes to check if fnic devcmd2 interface is exported by
the firmware. If devcmd2 interfaces is exported, driver starts using it
else falls back to fnic devcmd1 interface.
Signed-off-by: Satish Kharat <satishkh@cisco.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
Impose an upper limit on the max number of CQ entries (corresponding to the
copy wq) processed in an interrupt. Use module parameter to set the limit.
Signed-off-by: Satish Kharat <satishkh@cisco.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
Do RQ enable before posting descriptor. This is needed for later hw
revisions.
Signed-off-by: Satish Kharat <satishkh@cisco.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
Doing vnic_device_enable before this could cause interrupts to happen
before they are setup.
Signed-off-by: Satish Kharat <satishkh@cisco.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
This patch is to add fnic 20G port speed display in sysfs.
[mkp: typo]
Signed-off-by: Satish Kharat <satishkh@cisco.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
Most SCSI drivers want to enable "clustering", that is merging of
segments so that they might span more than a single page. Remove the
ENABLE_CLUSTERING define, and require drivers to explicitly set
DISABLE_CLUSTERING to disable this feature.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
Switch from the legacy PCI DMA API to the generic DMA API.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Johannes Thumshirn <jthumshirn@suse.de>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
This converts all remaining cases of the old setup_timer() API into using
timer_setup(), where the callback argument is the structure already
holding the struct timer_list. These should have no behavioral changes,
since they just change which pointer is passed into the callback with
the same available pointers after conversion. It handles the following
examples, in addition to some other variations.
Casting from unsigned long:
void my_callback(unsigned long data)
{
struct something *ptr = (struct something *)data;
...
}
...
setup_timer(&ptr->my_timer, my_callback, ptr);
and forced object casts:
void my_callback(struct something *ptr)
{
...
}
...
setup_timer(&ptr->my_timer, my_callback, (unsigned long)ptr);
become:
void my_callback(struct timer_list *t)
{
struct something *ptr = from_timer(ptr, t, my_timer);
...
}
...
timer_setup(&ptr->my_timer, my_callback, 0);
Direct function assignments:
void my_callback(unsigned long data)
{
struct something *ptr = (struct something *)data;
...
}
...
ptr->my_timer.function = my_callback;
have a temporary cast added, along with converting the args:
void my_callback(struct timer_list *t)
{
struct something *ptr = from_timer(ptr, t, my_timer);
...
}
...
ptr->my_timer.function = (TIMER_FUNC_TYPE)my_callback;
And finally, callbacks without a data assignment:
void my_callback(unsigned long data)
{
...
}
...
setup_timer(&ptr->my_timer, my_callback, 0);
have their argument renamed to verify they're unused during conversion:
void my_callback(struct timer_list *unused)
{
...
}
...
timer_setup(&ptr->my_timer, my_callback, 0);
The conversion is done with the following Coccinelle script:
spatch --very-quiet --all-includes --include-headers \
-I ./arch/x86/include -I ./arch/x86/include/generated \
-I ./include -I ./arch/x86/include/uapi \
-I ./arch/x86/include/generated/uapi -I ./include/uapi \
-I ./include/generated/uapi --include ./include/linux/kconfig.h \
--dir . \
--cocci-file ~/src/data/timer_setup.cocci
@fix_address_of@
expression e;
@@
setup_timer(
-&(e)
+&e
, ...)
// Update any raw setup_timer() usages that have a NULL callback, but
// would otherwise match change_timer_function_usage, since the latter
// will update all function assignments done in the face of a NULL
// function initialization in setup_timer().
@change_timer_function_usage_NULL@
expression _E;
identifier _timer;
type _cast_data;
@@
(
-setup_timer(&_E->_timer, NULL, _E);
+timer_setup(&_E->_timer, NULL, 0);
|
-setup_timer(&_E->_timer, NULL, (_cast_data)_E);
+timer_setup(&_E->_timer, NULL, 0);
|
-setup_timer(&_E._timer, NULL, &_E);
+timer_setup(&_E._timer, NULL, 0);
|
-setup_timer(&_E._timer, NULL, (_cast_data)&_E);
+timer_setup(&_E._timer, NULL, 0);
)
@change_timer_function_usage@
expression _E;
identifier _timer;
struct timer_list _stl;
identifier _callback;
type _cast_func, _cast_data;
@@
(
-setup_timer(&_E->_timer, _callback, _E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, &_callback, _E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, _callback, (_cast_data)_E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, &_callback, (_cast_data)_E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, (_cast_func)_callback, _E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, (_cast_func)&_callback, _E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, (_cast_func)_callback, (_cast_data)_E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, (_cast_func)&_callback, (_cast_data)_E);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E._timer, _callback, (_cast_data)_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, _callback, (_cast_data)&_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, &_callback, (_cast_data)_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, &_callback, (_cast_data)&_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, (_cast_func)_callback, (_cast_data)_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, (_cast_func)_callback, (_cast_data)&_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, (_cast_func)&_callback, (_cast_data)_E);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, (_cast_func)&_callback, (_cast_data)&_E);
+timer_setup(&_E._timer, _callback, 0);
|
_E->_timer@_stl.function = _callback;
|
_E->_timer@_stl.function = &_callback;
|
_E->_timer@_stl.function = (_cast_func)_callback;
|
_E->_timer@_stl.function = (_cast_func)&_callback;
|
_E._timer@_stl.function = _callback;
|
_E._timer@_stl.function = &_callback;
|
_E._timer@_stl.function = (_cast_func)_callback;
|
_E._timer@_stl.function = (_cast_func)&_callback;
)
// callback(unsigned long arg)
@change_callback_handle_cast
depends on change_timer_function_usage@
identifier change_timer_function_usage._callback;
identifier change_timer_function_usage._timer;
type _origtype;
identifier _origarg;
type _handletype;
identifier _handle;
@@
void _callback(
-_origtype _origarg
+struct timer_list *t
)
{
(
... when != _origarg
_handletype *_handle =
-(_handletype *)_origarg;
+from_timer(_handle, t, _timer);
... when != _origarg
|
... when != _origarg
_handletype *_handle =
-(void *)_origarg;
+from_timer(_handle, t, _timer);
... when != _origarg
|
... when != _origarg
_handletype *_handle;
... when != _handle
_handle =
-(_handletype *)_origarg;
+from_timer(_handle, t, _timer);
... when != _origarg
|
... when != _origarg
_handletype *_handle;
... when != _handle
_handle =
-(void *)_origarg;
+from_timer(_handle, t, _timer);
... when != _origarg
)
}
// callback(unsigned long arg) without existing variable
@change_callback_handle_cast_no_arg
depends on change_timer_function_usage &&
!change_callback_handle_cast@
identifier change_timer_function_usage._callback;
identifier change_timer_function_usage._timer;
type _origtype;
identifier _origarg;
type _handletype;
@@
void _callback(
-_origtype _origarg
+struct timer_list *t
)
{
+ _handletype *_origarg = from_timer(_origarg, t, _timer);
+
... when != _origarg
- (_handletype *)_origarg
+ _origarg
... when != _origarg
}
// Avoid already converted callbacks.
@match_callback_converted
depends on change_timer_function_usage &&
!change_callback_handle_cast &&
!change_callback_handle_cast_no_arg@
identifier change_timer_function_usage._callback;
identifier t;
@@
void _callback(struct timer_list *t)
{ ... }
// callback(struct something *handle)
@change_callback_handle_arg
depends on change_timer_function_usage &&
!match_callback_converted &&
!change_callback_handle_cast &&
!change_callback_handle_cast_no_arg@
identifier change_timer_function_usage._callback;
identifier change_timer_function_usage._timer;
type _handletype;
identifier _handle;
@@
void _callback(
-_handletype *_handle
+struct timer_list *t
)
{
+ _handletype *_handle = from_timer(_handle, t, _timer);
...
}
// If change_callback_handle_arg ran on an empty function, remove
// the added handler.
@unchange_callback_handle_arg
depends on change_timer_function_usage &&
change_callback_handle_arg@
identifier change_timer_function_usage._callback;
identifier change_timer_function_usage._timer;
type _handletype;
identifier _handle;
identifier t;
@@
void _callback(struct timer_list *t)
{
- _handletype *_handle = from_timer(_handle, t, _timer);
}
// We only want to refactor the setup_timer() data argument if we've found
// the matching callback. This undoes changes in change_timer_function_usage.
@unchange_timer_function_usage
depends on change_timer_function_usage &&
!change_callback_handle_cast &&
!change_callback_handle_cast_no_arg &&
!change_callback_handle_arg@
expression change_timer_function_usage._E;
identifier change_timer_function_usage._timer;
identifier change_timer_function_usage._callback;
type change_timer_function_usage._cast_data;
@@
(
-timer_setup(&_E->_timer, _callback, 0);
+setup_timer(&_E->_timer, _callback, (_cast_data)_E);
|
-timer_setup(&_E._timer, _callback, 0);
+setup_timer(&_E._timer, _callback, (_cast_data)&_E);
)
// If we fixed a callback from a .function assignment, fix the
// assignment cast now.
@change_timer_function_assignment
depends on change_timer_function_usage &&
(change_callback_handle_cast ||
change_callback_handle_cast_no_arg ||
change_callback_handle_arg)@
expression change_timer_function_usage._E;
identifier change_timer_function_usage._timer;
identifier change_timer_function_usage._callback;
type _cast_func;
typedef TIMER_FUNC_TYPE;
@@
(
_E->_timer.function =
-_callback
+(TIMER_FUNC_TYPE)_callback
;
|
_E->_timer.function =
-&_callback
+(TIMER_FUNC_TYPE)_callback
;
|
_E->_timer.function =
-(_cast_func)_callback;
+(TIMER_FUNC_TYPE)_callback
;
|
_E->_timer.function =
-(_cast_func)&_callback
+(TIMER_FUNC_TYPE)_callback
;
|
_E._timer.function =
-_callback
+(TIMER_FUNC_TYPE)_callback
;
|
_E._timer.function =
-&_callback;
+(TIMER_FUNC_TYPE)_callback
;
|
_E._timer.function =
-(_cast_func)_callback
+(TIMER_FUNC_TYPE)_callback
;
|
_E._timer.function =
-(_cast_func)&_callback
+(TIMER_FUNC_TYPE)_callback
;
)
// Sometimes timer functions are called directly. Replace matched args.
@change_timer_function_calls
depends on change_timer_function_usage &&
(change_callback_handle_cast ||
change_callback_handle_cast_no_arg ||
change_callback_handle_arg)@
expression _E;
identifier change_timer_function_usage._timer;
identifier change_timer_function_usage._callback;
type _cast_data;
@@
_callback(
(
-(_cast_data)_E
+&_E->_timer
|
-(_cast_data)&_E
+&_E._timer
|
-_E
+&_E->_timer
)
)
// If a timer has been configured without a data argument, it can be
// converted without regard to the callback argument, since it is unused.
@match_timer_function_unused_data@
expression _E;
identifier _timer;
identifier _callback;
@@
(
-setup_timer(&_E->_timer, _callback, 0);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, _callback, 0L);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E->_timer, _callback, 0UL);
+timer_setup(&_E->_timer, _callback, 0);
|
-setup_timer(&_E._timer, _callback, 0);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, _callback, 0L);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_E._timer, _callback, 0UL);
+timer_setup(&_E._timer, _callback, 0);
|
-setup_timer(&_timer, _callback, 0);
+timer_setup(&_timer, _callback, 0);
|
-setup_timer(&_timer, _callback, 0L);
+timer_setup(&_timer, _callback, 0);
|
-setup_timer(&_timer, _callback, 0UL);
+timer_setup(&_timer, _callback, 0);
|
-setup_timer(_timer, _callback, 0);
+timer_setup(_timer, _callback, 0);
|
-setup_timer(_timer, _callback, 0L);
+timer_setup(_timer, _callback, 0);
|
-setup_timer(_timer, _callback, 0UL);
+timer_setup(_timer, _callback, 0);
)
@change_callback_unused_data
depends on match_timer_function_unused_data@
identifier match_timer_function_unused_data._callback;
type _origtype;
identifier _origarg;
@@
void _callback(
-_origtype _origarg
+struct timer_list *unused
)
{
... when != _origarg
}
Signed-off-by: Kees Cook <keescook@chromium.org>
|
|
Setting speed based on the vinc device parameter read during
linkup. Also adding support to display 25,40 and 100G
Signed-off-by: Satish Kharat <satishkh@cisco.com>
Signed-off-by: Sesidhar Baddela <sebaddel@cisco.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
Instead define the timeout behavior purely based on the host_template
eh_timed_out method and wire up the existing transport implementations
in the host templates. This also clears up the confusion that the
transport template method overrides the host template one, so some
drivers have to re-override the transport template one.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Hannes Reinecke <hare@suse.com>
Reviewed-by: Tyrel Datwyler <tyreld@linux.vnet.ibm.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
This patch changes the !blk-mq path to the same defaults as the blk-mq
I/O path by always enabling block tagging, and always using host wide
tags. We've had blk-mq available for a few releases so bugs with
this mode should have been ironed out, and this ensures we get better
coverage of over tagging setup over different configs.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Jens Axboe <axboe@kernel.dk>
Reviewed-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: James Bottomley <JBottomley@Odin.com>
|
|
Since we got rid of ordered tag support in 2010 the prime use case of
switching on and off ordered tags has been obsolete. The other function
of enabling/disabling tagging entirely has only been correctly implemented
by the 53c700 driver and isn't generally useful.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com
Reviewed-by: Hannes Reinecke <hare@suse.de>
|
|
Drop the now unused reason argument from the ->change_queue_depth method.
Also add a return value to scsi_adjust_queue_depth, and rename it to
scsi_change_queue_depth now that it can be used as the default
->change_queue_depth implementation.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Mike Christie <michaelc@cs.wisc.edu>
Reviewed-by: Hannes Reinecke <hare@suse.de>
|
|
All drivers use the implementation for ramping the queue up and down, so
instead of overloading the change_queue_depth method call the
implementation diretly if the driver opts into it by setting the
track_queue_depth flag in the host template.
Note that a few drivers validated the new queue depth in their
change_queue_depth method, but as we never go over the queue depth
set during slave_configure or the sysfs file this isn't nessecary
and can safely be removed.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Mike Christie <michaelc@cs.wisc.edu>
Reviewed-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Venkatesh Srinivas <venkateshs@google.com>
|
|
In fnic_dev_wait, Wait for finish to complete at least three times in two
seconds while loop before returning -ETIMEDOUT as sometime
schedule_timeout_uninterruptible takes more than two seconds to wake up.
- Increment fnic version from 1.6.0.11 to 1.6.0.12
Signed-off-by: Hiral Shah <hishah@cisco.com>
Signed-off-by: Sesidhar Baddela <sebaddel@cisco.com>
Signed-off-by: Anil Chintalapati <achintal@cisco.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
|
|
Now that we also get proper values in cmd->request->tag for untagged
commands, there is no need to force tagged_supported to on in drivers
that need host-wide tags.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Mike Christie <michaelc@cs.wisc.edu>
Reviewed-by: Hannes Reinecke <hare@suse.de>
|
|
Remove the tagged argument from scsi_adjust_queue_depth, and just let it
handle the queue depth. For most drivers those two are fairly separate,
given that most modern drivers don't care about the SCSI "tagged" status
of a command at all, and many old drivers allow queuing of multiple
untagged commands in the driver.
Instead we start out with the ->simple_tags flag set before calling
->slave_configure, which is how all drivers actually looking at
->simple_tags except for one worke anyway. The one other case looks
broken, but I've kept the behavior as-is for now.
Except for that we only change ->simple_tags from the ->change_queue_type,
and when rejecting a tag message in a single driver, so keeping this
churn out of scsi_adjust_queue_depth is a clear win.
Now that the usage of scsi_adjust_queue_depth is more obvious we can
also remove all the trivial instances in ->slave_alloc or ->slave_configure
that just set it to the cmd_per_lun default.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Mike Christie <michaelc@cs.wisc.edu>
Reviewed-by: Hannes Reinecke <hare@suse.de>
Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>
|
|
Allow a driver to ask for block layer tags by setting .use_blk_tags in the
host template, in which case it will always see a valid value in
request->tag, similar to the behavior when using blk-mq. This means even
SCSI "untagged" commands will now have a tag, which is especially useful
when using a host-wide tag map.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Mike Christie <michaelc@cs.wisc.edu>
Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>
Reviewed-by: Hannes Reinecke <hare@suse.de>
|
|
Most drivers use exactly the same implementation, so provide it as a
library function.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Mike Christie <michaelc@cs.wisc.edu>
Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>
Reviewed-by: Hannes Reinecke <hare@suse.de>
|
|
Fnic Ctlr Path Trace utility is a tracing functionality built directly into fnic
driver to trace the control path frames like discovery, FLOGI request/reply,
PLOGI request/reply, link event etc. It will be one trace file for all fnics.
It will help us to debug and resolve the discovery and initialization related
issues in more convenient way. This trace information includes time stamp,
Host Number, Frame type, Frame Length and Frame. By default,64 pages are
allocated but we can change the number of allocated pages by module parameter
fnic_fc_trace_max_page. Each entry is of 256 byte and available entries are
depends on allocated number of pages. We can turn on or off the fnic control
path trace functionality by module paramter fc_trace_enable and/or reset the
trace contain by module paramter fc_trace_clear.
Signed-off-by: Hiral Shah <hishah@cisco.com>
Signed-off-by: Sesidhar Baddela <sebaddel@cisco.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
|
|
This patch contains following three minor fixes.
1) During Probe, fnic was sending FIP solicitation in Non FIP mode which is not
expected, setting the internal fip state to Non FIP mode explicitly, avoids
sending FIP frame.
2) When target goes offline, all outstanding IOs belong to the target will be
terminated by driver, If the termination count is high, then it influences
firmware responsiveness. To improve the responsiveness, default IO throttle
count is reduced to 256.
3) Accessing Virtual Fabric Id (vfid) and fc_map of Fibre-Channel Forwarder(FCF)
is invalid in fnic driver when Clear Virtual Link(CVL) is received prior to
receiving flogi reject from switch. As CVL clears all FCFs.
Signed-off-by: Hiral Shah <hishah@cisco.com>
Signed-off-by: Sesidhar Baddela <sebaddel@cisco.com>
Signed-off-by: Narsimhulu Musini <nmusini@cisco.com>
Signed-off-by: Anantha Tungarakodi <atungara@cisco.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial
Pull trivial tree updates from Jiri Kosina:
"Usual earth-shaking, news-breaking, rocket science pile from
trivial.git"
* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial: (23 commits)
doc: usb: Fix typo in Documentation/usb/gadget_configs.txt
doc: add missing files to timers/00-INDEX
timekeeping: Fix some trivial typos in comments
mm: Fix some trivial typos in comments
irq: Fix some trivial typos in comments
NUMA: fix typos in Kconfig help text
mm: update 00-INDEX
doc: Documentation/DMA-attributes.txt fix typo
DRM: comment: `halve' -> `half'
Docs: Kconfig: `devlopers' -> `developers'
doc: typo on word accounting in kprobes.c in mutliple architectures
treewide: fix "usefull" typo
treewide: fix "distingush" typo
mm/Kconfig: Grammar s/an/a/
kexec: Typo s/the/then/
Documentation/kvm: Update cpuid documentation for steal time and pv eoi
treewide: Fix common typo in "identify"
__page_to_pfn: Fix typo in comment
Correct some typos for word frequency
clk: fixed-factor: Fix a trivial typo
...
|
|
This feature gathers active and cumulative per fnic stats for io,
abort, terminate, reset, vlan discovery path and it also includes
various important stats for debugging issues. It also provided
debugfs and ioctl interface for user to retrieve these stats.
It also provides functionality to reset cumulative stats through
user interface.
Signed-off-by: Hiral Patel <hiralpat@cisco.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
|
|
Since commit 0998d0631001288a5974afc0b2a5f568bcdecb4d
(device-core: Ensure drvdata = NULL when no driver is bound),
the driver core clears the driver data to NULL after device_release
or on probe failure. Thus, it is not needed to manually clear the
device driver data to NULL.
Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Cc: James Bottomley <JBottomley@parallels.com>
Signed-off-by: Jiri Kosina <jkosina@suse.cz>
|
|
Introduced module params to provide dynamic way of configuring
queue depth.
Added support to get max io throttle count through UCSM to
configure maximum outstanding IOs supported by fnic and push
that value to scsi mid-layer.
Supported IO throttle values:
UCSM IO THROTTLE VALUE FNIC MAX OUTSTANDING IOS
------------------------------------------------------
16 (Default) 2048
<= 256 256
> 256 <ucsm value>
Signed-off-by: Hiral Patel <hiralpat@cisco.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
|
|
Issue was seen when SCSI buffer address is more than 40 bits in system
with more than 1.1TB RAM. When SCSI buffer is passed to VIC, it is failing
to map to correct buffer address, as DMA mask is set to 40 bits in driver
initialization. Corrected DMA_MASK from 40-bits to 64-bits to avoid masking
41-64 bits addresses.
Signed-off-by: Brian Uchino <buchino@cisco.com>
Signed-off-by: Hiral Patel <hiralpat@cisco.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
|
|
Code to reset fc_host statistics.
echo 1 > /sys/class/fc_host/hostX/statistics/reset_statistics clears fc_host stats,
the code also issues command to fnic firmware to clear vnic stats.
Signed-off-by: Narsimhulu Musini <nmusini@cisco.com>
Signed-off-by: Hiral Patel <hiralpat@cisco.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
|
|
I hit this during driver probe with the latest fnic updates (this trace
is from a backport into a distro kernel, but the issue is the same).
> BUG: sleeping function called from invalid context at mm/slab.c:3113
> in_atomic(): 0, irqs_disabled(): 1, pid: 610, name: work_for_cpu
> INFO: lockdep is turned off.
> irq event stamp: 0
> hardirqs last enabled at (0): [<(null)>] (null)
> hardirqs last disabled at (0): [<ffffffff81070aa5>]
> copy_process+0x5e5/0x1670
> softirqs last enabled at (0): [<ffffffff81070aa5>]
> copy_process+0x5e5/0x1670
> softirqs last disabled at (0): [<(null)>] (null)
> Pid: 610, comm: work_for_cpu Not tainted
> Call Trace:
> [<ffffffff810b2d10>] ? print_irqtrace_events+0xd0/0xe0
> [<ffffffff8105c1a7>] ? __might_sleep+0xf7/0x130
> [<ffffffff81184efb>] ? kmem_cache_alloc_trace+0x20b/0x2d0
> [<ffffffff8109709e>] ? __create_workqueue_key+0x3e/0x1d0
> [<ffffffff8109709e>] ? __create_workqueue_key+0x3e/0x1d0
> [<ffffffffa00c101c>] ? fnic_probe+0x977/0x11aa [fnic]
> [<ffffffffa00c1048>] ? fnic_probe+0x9a3/0x11aa [fnic]
> [<ffffffff81096f00>] ? do_work_for_cpu+0x0/0x30
> [<ffffffff812c6da7>] ? local_pci_probe+0x17/0x20
> [<ffffffff81096f18>] ? do_work_for_cpu+0x18/0x30
> [<ffffffff8109cdc6>] ? kthread+0x96/0xa0
> [<ffffffff8100c1ca>] ? child_rip+0xa/0x20
> [<ffffffff81550f80>] ? _spin_unlock_irq+0x30/0x40
> [<ffffffff8100bb10>] ? restore_args+0x0/0x30
> [<ffffffff8109cd30>] ? kthread+0x0/0xa0
> [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
The problem is in this hunk of "FIP VLAN Discovery Feature Support"
(d3c995f1dcf938f1084388d92b8fb97bec366566)
create_singlethreaded_workqueue cannot be called with irqs disabled
@@ -620,7 +634,29 @@ static int __devinit fnic_probe(struct pci_dev
*pdev,
vnic_dev_packet_filter(fnic->vdev, 1, 1, 0, 0, 0);
vnic_dev_add_addr(fnic->vdev, FIP_ALL_ENODE_MACS);
vnic_dev_add_addr(fnic->vdev, fnic->ctlr.ctl_src_addr);
+ fnic->set_vlan = fnic_set_vlan;
fcoe_ctlr_init(&fnic->ctlr, FIP_MODE_AUTO);
+ setup_timer(&fnic->fip_timer, fnic_fip_notify_timer,
+ (unsigned long)fnic);
+ spin_lock_init(&fnic->vlans_lock);
+ INIT_WORK(&fnic->fip_frame_work, fnic_handle_fip_frame);
+ INIT_WORK(&fnic->event_work, fnic_handle_event);
+ skb_queue_head_init(&fnic->fip_frame_queue);
+ spin_lock_irqsave(&fnic_list_lock, flags);
+ if (!fnic_fip_queue) {
+ fnic_fip_queue =
+ create_singlethread_workqueue("fnic_fip_q");
+ if (!fnic_fip_queue) {
+ spin_unlock_irqrestore(&fnic_list_lock, flags);
+ printk(KERN_ERR PFX "fnic FIP work queue "
+ "create failed\n");
+ err = -ENOMEM;
+ goto err_out_free_max_pool;
+ }
+ }
+ spin_unlock_irqrestore(&fnic_list_lock, flags);
+ INIT_LIST_HEAD(&fnic->evlist);
+ INIT_LIST_HEAD(&fnic->vlans);
} else {
shost_printk(KERN_INFO, fnic->lport->host,
"firmware uses non-FIP mode\n");
The attempts to make fnic_fip_queue a single instance for the driver
while it's being created in probe look awkward anyway, why is this not
created in fnic_init_module like the event workqueue?
Signed-off-by: Chris Leech <cleech@redhat.com>
Tested-by: Anantha Tungarakodi <atungara@cisco.com>
Acked-by: Hiral Patel <hiralpat@cisco.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
|
|
FIP VLAN discovery discovers the FCoE VLAN that will be used by all other FIP
protocols as well as by the FCoE encapsulation for Fibre Channel payloads on
the established virtual link. One of the goals of FC-BB-5 was to be as
nonintrusive as possible on initiators and targets, and therefore FIP VLAN
discovery occurs in the native VLAN used by the initiator or target to
exchange Ethernet traffic. The FIP VLAN discovery protocol is the only FIP
protocol running on the native VLAN; all other FIP protocols run on the
discovered FCoE VLANs.
If an administrator has manually configured FCoE VLANs on ENodes and FCFs,
there is no need to use this protocol. FIP and FCoE will run over the
configured VLANs.
An ENode without FCoE VLANs configuration would use this automated discovery
protocol to discover over which VLANs FCoE is running.
The ENode sends a FIP VLAN discovery request to a multicast MAC address called
All-FCF-MACs, which is a multicast MAC address to which all FCFs listen.
All FCFs that can be reached in the native VLAN of the ENode are expected to
respond on the same VLAN with a response that lists one or more FCoE VLANs
that are available for the ENode's VN_Port login. This protocol has the sole
purpose of allowing the ENode to discover all the available FCoE VLANs.
Now the ENode may enable a subset of these VLANs for FCoE Running the FIP
protocol in these VLANs on a per VLAN basis. And FCoE data transactions also
would occur on this VLAN. Hence, Except for FIP VLAN discovery, all other FIP
and FCoE traffic runs on the selected FCoE VLAN. Its only the FIP VLAN
Discovery protocol that is permitted to run on the Default native VLAN of the
system.
[**** NOTE ****]
We are working on moving this feature definitions and functionality to libfcoe
module. We need this patch to be approved, as Suse is looking forward to merge
this feature in SLES 11 SP3 release. Once this patch is approved, we will
submit patch which should move vlan discovery feature to libfoce.
[Fengguang Wu <fengguang.wu@intel.com>: kmalloc cast removal]
Signed-off-by: Anantha Prakash T <atungara@cisco.com>
Signed-off-by: Hiral Patel <hiralpat@cisco.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
|
|
Fnic Trace utility is a tracing functionality built directly into fnic driver
to trace events. The benefit that trace buffer brings to fnic driver is the
ability to see what it happening inside the fnic driver. It also provides the
capability to trace every IO event inside fnic driver to debug panics, hangs
and potentially IO corruption issues. This feature makes it easy to find
problems in fnic driver and it also helps in tracking down strange bugs in a
more manageable way. Trace buffer is shared across all fnic instances for
this implementation.
Signed-off-by: Hiral Patel <hiralpat@cisco.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
|
|
1. Handling overlapped firmware resets
This fix serialize multiple firmware resets to avoid situation where fnic
device fails to come up for link up event, when firmware resets are issued
back to back. If there are overlapped firmware resets are issued,
the firmware reset operation checks whether there is any firmware reset in
progress, if so it polls for its completion in a loop with 100ms delay.
2. Handling device reset timeout
fnic_device_reset code has been modified to handle Device reset timeout:
- Issue terminate on device reset timeout.
- Introduced flags field (one of the scratch fields in scsi_cmnd).
With this, device reset request would have DEVICE_RESET flag set for other
routines to determine the type of the request.
Also modified fnic_terminate_rport_io, fnic_rport_exch_rset, completion
routines to handle SCSI commands with DEVICE_RESET flag.
3. LUN/Device Reset hangs when issued through IOCTL using utilities like
sg_reset.
Each SCSI command is associated with a valid tag, fnic uses this tag to
retrieve associated scsi command on completion. the LUN/Device Reset issued
through IOCTL resulting into a SCSI command that is not associated with a
valid tag. So fnic fails to retrieve associated scsi command on completion,
which causes hang. This fix allocates tag, associates it with the
scsi command and frees the tag, when the operation completed.
4. Preventing IOs during firmware reset.
Current fnic implementation allows IO submissions during firmware reset.
This fix synchronizes IO submissions and firmware reset operations.
It ensures that IOs issued to fnic prior to reset will be issued to the
firmware before firmware reset.
Signed-off-by: Narsimhulu Musini <nmusini@cisco.com>
Signed-off-by: Hiral Patel <hiralpat@cisco.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
|
|
CONFIG_HOTPLUG is going away as an option. As a result, the __dev*
markings need to be removed.
This change removes the use of __devinit, __devexit_p, __devinitdata,
__devinitconst, and __devexit from these drivers.
Based on patches originally written by Bill Pemberton, but redone by me
in order to handle some of the coding style issues better, by hand.
Cc: Bill Pemberton <wfp5p@virginia.edu>
Cc: Adam Radford <linuxraid@lsi.com>
Cc: "James E.J. Bottomley" <JBottomley@parallels.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
|
|
Driver was incorrectly using the SLAB_CACHE_DMA flag when creating a cache
for SGLs. fnic device does not have 24-bit DMA restrictions. Remove the flag
and allocations from ZONE_DMA.
Thanks to Roland Dreier and David Rientjes for pointing out the bug.
Signed-off-by: Abhijeet Joglekar <abjoglek@cisco.com>
Signed-off-by: James Bottomley <JBottomley@Parallels.com>
|
|
This removes the driver's get_host_def_dev_loss_tmo
callback and just has the driver set the dev loss
using the fc class fc_host_dev_loss_tmo macro like is
done for other fc params.
This also adds a set rport dev loss function so the
fc class host dev loss tmp sysfs support being added
in the fc class patch can update rports.
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
|
|
This fixes a bug where the driver was resetting the
rport dev_loss_tmo when devices were added by adding
support for the get_host_def_dev_loss_tmo callout.
Patch has only been compile tested.
Signed-off-by: Mike Christie <michaelc@cs.wisc.edu>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
|
|
The FC-BB-6 committee is proposing a new FIP usage model called
VN_port to VN_port mode. It allows VN_ports to discover each other
over a loss-free L2 Ethernet without any FCF or Fibre-channel fabric
services. This is point-to-multipoint. There is also a variant
of this called point-to-point which provides for making sure there
is just one pair of ports operating over the Ethernet fabric.
We add these new states: VNMP_START, _PROBE1, _PROBE2, _CLAIM, and _UP.
These usually go quickly in that sequence. After waiting a random
amount of time up to 100 ms in START, we select a pseudo-random
proposed locally-unique port ID and send out probes in states PROBE1
and PROBE2, 100 ms apart. If no probe responses are heard, we
proceed to CLAIM state 400 ms later and send a claim notification.
We wait another 400 ms to receive claim responses, which give us
a list of the other nodes on the network, including their FC-4
capabilities. After another 400 ms we go to VNMP_UP state and
should start interoperating with any of the nodes for whic we
receivec claim responses. More details are in the spec.j
Add the new mode as FIP_MODE_VN2VN. The driver must specify
explicitly that it wants to operate in this mode. There is
no automatic detection between point-to-multipoint and fabric
mode, and the local port initialization is affected, so it isn't
anticipated that there will ever be any such automatic switchover.
It may eventually be possible to have both fabric and VN2VN
modes on the same L2 network, which may be done by two separate
local VN_ports (lports).
When in VN2VN mode, FIP replaces libfc's fabric-oriented discovery
module with its own simple code that adds remote ports as they
are discovered from incoming claim notifications and responses.
These hooks are placed by fcoe_disc_init().
A linear list of discovered vn_ports is maintained under the
fcoe_ctlr struct. It is expected to be short for now, and
accessed infrequently. It is kept under RCU for lock-ordering
reasons. The lport and/or rport mutexes may be held when we
need to lookup a fcoe_vnport during an ELS send.
Change fcoe_ctlr_encaps() to lookup the destination vn_port in
the list of peers for the destination MAC address of the
FIP-encapsulated frame.
Add a new function fcoe_disc_init() to initialize just the
discovery portion of libfcoe for VN2VN mode.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
|
|
There are three modes that libfcoe currently supports, and a new one
is coming. Change the fcoe_ctlr_init() interface to add the mode
desired. This should not change any functionality.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
|
|
No reason to restrict CDB size to 12 bytes in fcoe, so
increased to 16 so that 16 bytes SCSI CDB doesn't fail.
Uses common define to set max_cmd_len for fcoe and fnic,
fnic is already setting max_cmd_len to 16.
sg_readcap -l fails without this fix.
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
|
|
implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
|
|
Incorrect initialization of lport stats in fnic_probe() causes fnic to
crash at bootup and a node hang if fip is enabled and all links are brought
up after fnic is loaded.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Venkata Siva Vijayendra Bhamidipati <vbhamidi@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
|
|
To enable FIP support in fnic, we have to register with hardware to receive
FIP solication frames on a well-known multicast address.
Before FIP support, the firmware interface allowed multicast address
registrations only for enic devices. This is a minor change in fnic to
allow the firmware interface to now register mcast addresses for fnic too.
Signed-off-by: Brian Uchino <buchino@cisco.com>
Signed-off-by: Herman Lee <hermlee@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
|
|
Add initialization of .bsg_request in the scsi_transport_fc
template so that fcping works.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
|
|
Use libfcoe as a common FIP implementation with fcoe.
FIP or non-FIP mode is fully automatic if the firmware
supports and enables it.
Even if FIP is not supported, this uses libfcoe for the non-FIP
handling of FLOGI and its response.
Use the new lport_set_port_id() notification to capture
successful FLOGI responses and port_id resets.
While transitioning between Ethernet and FC mode, all rx and
tx FC frames are queued. In Ethernet mode, all frames are
passed to the exchange manager to capture FLOGI responses.
Change to set data_src_addr to the ctl_src_addr whenever it
would have previously been zero because we're not logged in.
This seems safer so we'll never send a frame with a 0 source MAC.
This also eliminates a special case for sending FLOGI frames.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
|