Bossini 4 Assignments

Assignment 4: Software Defined Networking

CS640 Spring 2016

Due:  Tuesday, April 19 at 11pm

Overview

For this project you will implement two control application for a software defined network (SDN). A layer-3 routing application will install rules in SDN switches to forward traffic to hosts using the shortest, valid path through the network. A distributed load balancer application will redirect new TCP connections to hosts in a round-robin order.

Part 1: Getting Started

Part 2: Layer-3 Routing Application

Part 3: Distributed Load Balancer Application

Submission Instructions

Learning Objectives

After completing this programming assignment, students should be able to:

  • Contrast SDN applications and traditional network control planes
  • Create SDN applications that use proactive or reactive flow installation

Clarifications

  • You should make the following edit to the L3Routing.java file we provided:
    1) Change the line this.knownHosts.remove(host) in the deviceRemoved(...) event handler to this.knownHosts.remove(device)

Part 1: Getting Started

Your SDN applications will be written in Java and run atop the Floodlight OpenFlow controller. You will use Mininet to emulate a variety of network topologies consisting of OpenFlow switches and hosts. You should continue to use your assigned Mininet virtual machine (VM) for this project.

Preparing Your Environment

Before beginning this project, there are some additional steps you need to complete to prepare your VM:

  1. Install required packages

sudo apt-get update
sudo apt-get install -y curl traceroute ant openjdk-7-jdk git iputils-arping

  1. Download Floodlight:
            cd ~
            git clone https://bitbucket.org/sdnhub/floodlight-plus.git
  2. Download the starter code: http://pages.cs.wisc.edu/~akella/CS640/S16/assignment4/assign4.tgz

cd ~
wget http://pages.cs.wisc.edu/~akella/CS640/S16/assignment4/assign4.tgz
tar xzvf assign4.tgz

  1. Symlink Floodlight
            cd ~/assign4/
            ln -s ../floodlight-plus
  2. Patch Floodlight
            cd ~/assign4/floodlight-plus
            patch -p1 < ~/assign4/floodlight.patch
  3. Install new version of Mininet and patch it

cd ~/assign4

./fixmininet.sh

If you want to use Eclipse (on a machine other than your VM) for development, you should do the following:

  1. cd floodlight-plus
    ant eclipse
  2. Open Eclipse and create a new workspace. Do not choose the floodlight-plus or assign4 folders as the location for your workspace.
  3. Select File → Import → General → Existing Projects into Workspace. Click Next.
  4. From Select root directory click Browse. Select the parent folder containing your floodlight-plus and assign4 folders.
  5. Check the boxes for floodlight-plus and assign4. Click Finish.
  6. Exclude the src/test/java folder from the build path:
  1. Right click on this folder in the Project Explorer pane.
  2. Select Build Path →Remove from Build Path.
  1. Add the floodlight-plus project to the build path of assign4:
  1. Right click assign4 in the Project Explorer pane.
  2. Select Build Path →Configure Build Path.
  3. Under the Projects tab; click Add.
  4. Check the floodlight-plus project; click OK.

Running Your Control Applications

  1. Compile Floodlight and your applications:

  cd ~/assign4/
 ant

This will produce a jar file FloodlightWithApps.jar that includes the compiled code for Floodlight and your SDN applications.

  1. Start Floodlight and your SDN applications:
      java -jar FloodlightWithApps.jar -cf l3routing.prop
    The above command will start Floodlight and only your layer-3 routing application. You can start both your layer-3 routing and load balancer applications by using loadbalancer.prop for the -cf (configuration file) argument.

When Floodlight starts, you should see output like the following:

19:15:37.239 INFO [n.f.c.m.FloodlightModuleLoader:main] Loading modules from file l3routing.prop

19:15:37.666 INFO [n.f.c.i.Controller:main] Controller role set to MASTER

19:15:37.676 INFO [n.f.c.i.Controller:main] Flush switches on reconnect -- Disabled

19:15:37.697 INFO [ArpServer:main] Initializing ArpServer...

19:15:37.697 INFO [L3Routing:main] Initializing L3Routing...

19:15:38.835 INFO [n.f.l.i.LinkDiscoveryManager:main] Setting autoportfast feature to OFF

19:15:38.891 INFO [ArpServer:main] Starting ArpServer...

19:15:38.892 INFO [L3Routing:main] Starting L3Routing...

19:15:39.122 INFO [o.s.s.i.c.FallbackCCProvider:main] Cluster not yet configured; using fallback local configuration

19:15:39.122 INFO [o.s.s.i.SyncManager:main] [32767] Updating sync configuration ClusterConfig [allNodes={32767=Node [hostname=localhost, port=6642, nodeId=32767, domainId=32767]}, authScheme=NO_AUTH, keyStorePath=null, keyStorePassword is unset]

19:15:39.185 INFO [o.s.s.i.r.RPCService:main] Listening for internal floodlight RPC on localhost/127.0.0.1:6642

19:15:39.421 INFO [n.f.c.i.Controller:main] Listening for switch connections on 0.0.0.0/0.0.0.0:6633

Keep the terminal with Floodlight open, as you will need to see the output for debugging. Use another terminal for the next step. Now is a time to use screen. You can install screen in your VM by running: sudo apt-get install screen.

  1. Start Mininet:
      sudo ./run_mininet.py single,3
    The above command will create a topology with a single SDN switch (s1) and three hosts (h1 - h3) directly connected to the switch:

    You can change the number of hosts by changing the numeric value included in the arguments to the run_mininet.py script. You can also start Mininet with six other topologies:

Once mininet has started, you should see Floodlight produce output like the following:

11:04:44.960 INFO [n.f.c.i.OFChannelHandler:New I/O server worker #2-1] New switch connection from /127.0.0.1:50921

11:04:44.999 INFO [n.f.c.i.OFChannelHandler:New I/O server worker #2-1] Disconnected switch [/127.0.0.1:50921 DPID[?]]

11:04:46.278 INFO [n.f.c.i.OFChannelHandler:New I/O server worker #2-2] New switch connection from /127.0.0.1:50922

11:04:46.355 INFO [n.f.c.i.OFChannelHandler:New I/O server worker #2-2] Switch OFSwitchBase [/127.0.0.1:50922 DPID[00:00:00:00:00:00:00:01]] bound to class class net.floodlightcontroller.core.internal.OFSwitchImpl, writeThrottle=false, description OFDescriptionStatistics [Vendor: Nicira, Inc., Model: Open vSwitch, Make: None, Version: 2.0.1, S/N: None]

11:04:46.364 INFO [n.f.c.OFSwitchBase:New I/O server worker #2-2] Clearing all flows on switch OFSwitchBase [/127.0.0.1:50922 DPID[00:00:00:00:00:00:00:01]]

11:04:46.377 WARN [n.f.c.i.C.s.notification:main] Switch 00:00:00:00:00:00:00:01 connected.

11:04:46.383 INFO [L3Routing:main] Switch s1 added

11:04:46.432 INFO [L3Routing:Topology Updates] Link s1:0 -> host updated

11:04:46.484 INFO [L3Routing:Topology Updates] Link s1:2 -> host updated

11:04:46.489 WARN [n.f.c.i.C.s.notification:main] Switch 00:00:00:00:00:00:00:01 port s1-eth1 changed: UP

11:04:46.585 INFO [L3Routing:Topology Updates] Link s1:1 -> host updated

  1. You can now run commands (e.g., ping) in mininet like you did for the previous assignments. However, until you complete Part 2, no packets will be forwarded.

You should always start Floodlight and your SDN applications before starting mininet. Also, we recommend that you restart Floodlight and your SDN applications whenever you restart mininet.

Resources

If you have questions about SDN, OpenFlow, or Floodlight you may want to consult:


Part 2: Layer-3 Routing Application

Background

In Assignment 3, you wrote a router control plane that used RIP to construct a route table. The route table was then used by your router’s data plane to determine where (i.e., out which interface) a packet should be forwarded.

Your first SDN application will also construct route tables. However, rather than exchanging RIP messages between routers, your application will construct route tables based on a global view of the network topology. The appropriate route table will be installed in each SDN switch by your application, and each SDN switch will forward packets according to the installed route table.

It is important to note that SDN switches do not directly support the route table structure used in the last programming assignment. Rather, SDN switches use a more general flow table structure, which can accommodate route tables (used in traditional layer-3 routers), MAC learning tables (used in traditional layer-2 switches), and many other constructs. Each entry, or rule, in a flow table has match criteria that defines (on the basis of fields in Ethernet, IP, TCP, UDP, and other headers) which packets the rule applies to. Each entry also has one or more instructions/actions which should be taken for each packet that matches the rule.

Your layer-3 routing application will install entries that match packets based on their destination IP address (and Ethernet type), and execute an output action to send the packet out a specific port on the SDN switch.

(You’ll use other match criteria and additional instructions/actions for the other SDN application you write, which is described in Part 3.) The match criteria serves the same purpose as the destination and mask fields in a traditional route table, and the output action servers the same purpose as the interface field in a traditional route table. There is no concept of a gateway in SDN flow tables, but that’s okay—your router only used the gateway to determine how to rewrite a packet’s destination MAC address to ensure correct layer-2 forwarding, and we aren’t using traditional layer-2 forwarding in SDN.

Code Overview

The code for your layer-3 routing application will reside in L3Routing.java in the edu.wisc.cs.sdn.apps.l3routing package. The file we provided already contains code to:

  • Access host and topology information from other modules (or applications) included with Floodlight—see the getHosts(), getSwitches(), and getLinks() methods
  • Receive notifications about changes in the network—deviceAdded(...), deviceRemoved(...), deviceMoved(...), switchAdded(...), switchRemoved(...), and linkDiscoveryUpdate(...)

We have also provided code in the edu.wisc.cs.sdn.apps.utils package for:

  • An SDN application that responds to ARP requests from hosts—see ArpServer.java
  • Telling a switch to install a rule in the flow table, remove rules from the flow table, and send a packet—see SwitchCommands.java

TODOs

You need to complete the TODOs in L3Routing.java to install and remove flow table entries from SDN switches such that traffic is forwarded to a host using the shortest path. Note that Floodlight creates only one instance (i.e., object) of the L3Routing class; this instances handles all switches.

Computing Shortest Paths

You should use the Bellman-Ford algorithm to compute the shortest paths to reach a host h from every other host h’ ∈ H, h ≠ h’ (H is the set of all hosts). You can use the getHosts(), getSwitches(), and getLinks() methods to get the topology information that you need to provide as input to the Bellman-Ford algorithm.


There will be two link objects between pairs of switches, one in each direction. Due to the way links are discovered, there may be a short period of time (tens of milliseconds) where the controller has a link object only in one direction. If you have a link object in one direction, you can assume the physical link is bidirectional.

When a host joins the network, both the deviceAdded(...) and linkDiscoveryUpdate(...) event handlers will be called. There are no guarantees on which order these event handlers are called.  Thus, a host may be added but we may not yet know which switch it is linked to. The isAttachedToSwitch() method in the Host class will return true if we know the switch to which a host is connected, otherwise it will return false. If the method returns false, then you do not need to install any rules to route traffic to this host.

You can assume the following will always hold true in the network:

  • The network is a connected graph.  In other words, there will always be at least one possible path between every pair of switches.
  • There is only one physical link between a pair of switches.
  • Links are undirected. However, be aware that Floodlight maintains a Link object for each direction (i.e., there are two Link objects for each physical link).

Installing Rules

Once you have determined the shortest path to reach host h from h’, you must install a rule in the flow table in every switch in the path. The rule should match IP packets (i.e., Ethernet type is IPv4) whose destination IP is the IP address assigned to host h. You can specify this in Floodlight by creating a new OFMatch object and calling the set methods for the appropriate fields; you must set the Ethernet Type before you set the destination IP. The rule’s action should be to output packets on the appropriate port in order to reach the next switch in the path. You can specify this in Floodlight by creating an OFInstructionApplyActions object whose set of actions consists of a single OFActionOutput object with the appropriate port number.

SDN switches have multiple flow tables—we discuss this more in Part 3. For now, you should install rules in the table specified in the table class variable in the L3Routing class. Also, your rules should never timeout and have a default priority (both defined as constants in the SwitchCommands class).

Testing and Debugging

You should test your code by sending traffic between various hosts in the network topology—mininet’s built-in pingall command is very useful for this.

To help you debug, you can view the contents of an SDN switch’s flow tables by running the following command in your mininet VM (not in mininet itself):

sudo ovs-ofctl -O OpenFlow13 dump-flows s1

This will output the contents of s1’s flow tables. Change the last argument to output the flow tables from a different switch.

Triggering Event Handlers

  • You can trigger the linkDiscoveryUpdate(...) event handler by running any of the following commands in mininet (substituting switch and host names as desired):
  • link s1 s2 down — takes down the link between s1 and s2; you can assume the network is a connected graph, so you should never take down a link that would result in a disconnected graph
  • link s1 s2 up  — brings up the link between s1 and s2
  • link s1 h1 down  — takes down the link between s1 and h1; this will also result in a deviceRemoved(...) event and the isAttachedToSwitch() method for the Host object for h1 will now return false
  • link s1 h1 up  — brings up the link between s1 and h1; this will also result in a deviceMoved(...) event and the isAttachedToSwitch() method for the Host object for h1 will now return true
  • You can trigger the deviceRemoved(...) event handler by taking down a link between a switch and a host, as described above
  • You can trigger the deviceMoved(...) event handler by bringing up a link between a switch and a host, as described above
  • You can trigger the switchRemoved(...) event handler by running the following command in a regular terminal window (not in mininet):
      sudo ovs-vsctl del-br s1
    Note that once a switch is removed, you cannot easily add it back without restarting mininet. You can assume the network is a connected graph, so you should never remove a switch that would result in a disconnected graph.

Part 3: Distributed Load Balancer Application

Background

Networks employ load balancing to distribute client requests among a collection of hosts running a specific service (e.g., a web server). In class, we discussed how DNS could be used to implement load balancing. Load balancing is also commonly implemented using a special piece of hardware.

A hardware load balancer is placed in the network and configured with an IP address (e.g., 10.0.100.1) and a set of hosts among which it should distribute requests (e.g., 10.0.0.2 and 10.0.0.3). Clients wanting to communicate with a service (e.g., a web server) running on those hosts are provided with the IP address of the load balancer, not the IP address of a specific host. Clients initiate a TCP connection to the IP address of the load balancer (10.0.100.1) and the TCP port associated with the service (e.g., port 80).

For each new TCP connection, the load balancer selects one of the specified hosts (usually in round robin order). The load balancer maintains a mapping of active connections—identified by the client’s IP and TCP port—to the assigned hosts.

For all packets sent from clients to the load balancer, the load balancer rewrites the destination IP and MAC addresses to the IP and MAC addresses of the selected host. The mapping information stored by the load balancer is used to determine the appropriate host IP and MAC addresses that should be written into a packet arriving from a client. For all packets sent from servers to clients, the load balancer rewrites the source IP and MAC addresses to the IP and MAC addresses of the load balancer.

Your second SDN application will implement the same functionality as a set of hardware load balancers. Your application will be provided with a list of virtual IPs and a set of hosts among which connections to the virtual IPs should be load balanced. (We use the term virtual IP because the IP address is not actually assigned to any node in the network.) When clients initiate TCP connections with a specific virtual IP, SDN switches will send the TCP SYN packet to the SDN controller. Your SDN application will select a host from a pre-defined set, and install rules in an SDN switch to rewrite the IP and MAC addresses of packets associated with the connection. You will also instruct the SDN switch to match the modified packets against the flow rules installed by your layer-3 routing application and apply the appropriate actions (i.e., send the packets out the appropriate ports).

Code Overview

The code for your load balancer application will reside in LoadBalancer.java in the edu.wisc.cs.sdn.apps.loadbalancer package. The file we provided already contains code to:

  • Receive a notification when a switch joins the network—switchAdded(...)
  • Receive a packet from a switch when the packet did not match any entries in the switch’s flow table—receive(...)

The LoadBalancerInstance class represents a single distributed load balancer. (We use the term distributed because the load balancing is performed at many switches, rather than at a single hardware load balancer.) Each load balancer instance has a virtual IP address, virtual MAC address, and set of hosts among which TCP connections should be distributed. The instances class variable in the LoadBalancer class maps a virtual IP address to a specific load balancer instance.

TODOs

You need to complete the TODOs in LoadBalancer.java to:

  • Install rules in every switch (when it joins the network) to:
  • Notify the controller when a client initiates a TCP connection with a virtual IP—we cannot specify TCP flags in match criteria, so the SDN switch will notify the controller of each TCP packet sent to a virtual IP which did not match a connection-specific rule (described below)
  • Notify the controller when a client issues an ARP request for the MAC address associated with a virtual IP
  • Match all other packets against the rules in the next table in the switch (described below)
  • Install connection-specific rules for each new connection to a virtual IP to:
  • Rewrite the destination IP and MAC address of TCP packets sent from a client to the virtual IP
  • Rewrite the source IP and MAC address of TCP packets sent from server to client

Connection-specific rules should match packets on the basis of Ethernet type, source IP address, destination IP address, protocol, TCP source port, and TCP destination port. Connection-specific rules should take precedence over the rules that send TCP packets to the controller, otherwise every TCP packet would be sent to the controller. Therefore, these rules should have a higher priority than the rules installed when a switch joins the network.  Also, we want connection-specific rules to be removed when a TCP connection ends, so connection-specific rules should have an idle timeout of 20 seconds.

  • Construct and send an ARP reply packet when a client requests the MAC address associated with a virtual IP

Multiple Tables

Your load balancer application should work in tandem with your layer-3 routing application. To achieve this, you will need to leverage the multiple tables feature of OpenFlow switches. When packets first arrive at an OpenFlow switch, they are matched against the rules in table 0. The actions for these rules can specify that the packets be modified, output, sent to the controller, and/or matched against the rules in a different table.

Your load balancer application should install rules in the table specified in the table class variable in the LoadBalancer class—set to table 0 in the loadbalancer.prop configuration file. The connection-specific rules that modify IP and MAC addresses should include an instruction to match the modified packets against the rules installed by your layer-3 routing application. Since your layer-3 routing application will install rules in the table class variable in the L3Routing class, this instruction should direct packets to the table defined in this class variable. The modified packet will then be matched against these rules and forwarded out the appropriate port.

All packets which are not TCP packets destined for a virtual IP, or packets associated with a connection that has already been assigned to a specific host, should be send directly the table used by your layer-3 routing application.

Sending ARP Packets

When a client wants to initiate a connection with the virtual IP, it will need to determine the MAC address associated with the virtual IP using ARP.  The client does not know the IP is virtual, and since it’s not actually assigned to any host, your SDN application must take responsibility for replying to these requests.

You can construct an ARP reply packet using the classes in the net.floodlightcontroller.packet package. You can use the sendPacket(...) method in the SwitchCommands class to send the packet.

Rule Instructions/Actions

When a rule should send a packet to the controller, the rule should include an OFInstructionApplyActions whose set of actions consists of a single OFActionOutput with OFPort.OFPP_CONTROLLER as the port number.

When a rule should rewrite the destination IP and MAC addresses of a packet, the rule should include an OFInstructionApplyActions whose set of actions consists of:

  • An OFActionSetField with a field type of OFOXMFieldType.ETH_DST and the desired MAC address as the value
  • An OFActionSetField with a field type of OFOXMFieldType.IPV4_DST and the desired IP address as the value

The actions for rewriting the source IP and MAC addresses of a packet are similar.

When a packet should be processed by the SDN switch based on the rules installed by your layer-3 routing application, a rule should include an OFInstructionGotoTable whose table number is the value specified in the table class variable in the L3Routing class.

Testing and Debugging

You should test your code by issuing web requests (using curl) from a client host to the virtual IPs.

You can add or remove virtual IPs and hosts by modifying the loadbalancer.prop file.

To see which packets a host is sending/recieving run:

tcpdump -v -n -i hN-eth0

replacing N with the host’s number.


Submission Instructions

You must submit a single tar file of the src directory containing the Java source files for your SDN applications.  Please submit the entire src directory; do not submit any other files or directories. To create the tar file, run the following command, replacing username1 and username2 with the WISC username (NetID) of each group member:

tar czvf username1_username2.tgz src

Upload the tar file to the Assignment 4 dropbox on Learn @ UW. Please submit only one tar file per group.

 

Abstract

The results of an external quality-assessment experiment for serum creatinine measurement are described. Fifty-one laboratories performed quintuplicate analyses during three different analytical runs on six lyophilized sera and two frozen human serum pools. Isotope dilution gas chromatography–mass spectrometry (ID GC-MS) target values were assigned to all the materials. Intralaboratory within- and between-run imprecision results were very similar for all the materials tested (CV ≤2.20% and ≤4.70%, respectively). The overall imprecision obtained was high (CV 6.5–20.0%) because of increased interlaboratory–intermethod variability. A significant positive bias (+9.2–+43.7%) was found for all the materials at lower creatinine concentration. By using two human sera at different concentrations, we could calculate the constant and the proportional calibration bias displayed by each peer group. The majority of the lyophilized materials showed a behavior divergent from the frozen pools, indicating matrix-related problems. We propose a new algorithm for calculating matrix bias correction factor instrument–reagent specific for each material.

The analytical goals for creatinine on the basis of biological variability are very demanding (CV ≤2.2% for precision and ≤2.8% for accuracy (1)). Imprecision is strictly dependent on analyzer characteristics, and can be easily verified. On the contrary, variables affecting inaccuracy (method specificity, type of calibration, calibrator matrix, and value assignment) are more difficult to identify and control. These variables lead to a wide dispersion of results among different laboratories. As a result, the measurement appears far from the desirable performances.

Currently, there is more room for improvement in accuracy than for precision in creatinine determination. Through the use of reference methods and appropriate materials, it is possible to come closer to the “trueness” of the results. The availability of a definitive method is certainly a problem, but for accuracy of routine methods the real difficulty is the material used. The case of creatinine is particularly critical because the lack of commutability (2) is emphasized by the poor specificity and weakness of the majority of the routinely used picrate reaction-based methods (3)(4)(5).

This fact forces almost all the proficiency testing programs to use peer group target values without any means to verify the real accuracy of any single laboratory, and without progress toward improvement of the agreement between the different laboratories.

Here we describe the results of an external quality-assessment scheme (EQAS) from 51 laboratories of Lombardy region (Italy).1 We tried to focus on several aspects related to the accuracy of creatinine measurement. First, with a peculiar experimental design of replicate analyses, we could estimate components of variability. Second, through the use of frozen sera and the ultimate accuracy reference, an isotope dilution gas chromatography–mass spectrometry (ID GC-MS) method, we calculated constant and proportional components of the calibration error. Third, we verified the presence of matrix effects in most lyophilized sera and, with a modification of the algorithm proposed by Ross et al. (6), we calculated a matrix bias correction factor.

Materials and Methods

id gc-ms creatinine method

A Finnigan MAT95 mass spectrometer (Finnigan MAT, Bremen, Germany) was used for GC-MS analysis. HPLC purification was carried out with a Jasco HPLC pump (model PU880, Tokyo, Japan) and with a variable-wavelength ultraviolet detector (model 875-UV, Jasco). The HPLC column was a Lichrosphere 100 RP-18 (250 × 4 mm, 5-μm particles) from Merck (Darmstadt, Germany).

Creatinine [Standard Reference Material (SRM) 914a, 99.8% purity] and lyophilized reference sera SRM 909a1 (certified value = 84 ± 1 μmol/L) and 909a2 (463 ± 6 μmol/L) were from NIST (October 13, 1993; revision of certificate dated February 24, 1993). Sera were reconstituted according to the NIST insert. [2H3]Creatinine (98 atom % excess) was from Isotec (Miamisburg, OH). N-methyl- N-(tert-butyldimethylsilyl)-trifluoroacetamide (MTBSTFA) was purchased from Fluka (Buchs, Switzerland). All solvents and general chemicals used were of analytical grade.

All solutions and sera were dispensed with known accuracy and imprecision as already reported (7). Calibrators were prepared by mixing various amounts of SRM 914a creatinine with [2H3]creatinine to provide a series of mixtures with known ratios of the two isotopomers between 0.8–1.2.

Weighed amounts of each serum were supplemented with a weighed aliquot of the [2H3]creatinine solution to get about a 1:1 ratio of [1H]creatinine:[2H]creatinine. After stirring, supplemented sera were kept at room temperature for 2 h to allow equilibration before protein precipitation obtained with acetone. The aqueous phase was separated and evaporated to dryness under reduced pressure. An isocratic separation of creatine from creatinine was achieved by HPLC with H2O containing 0.1% HCOOH (pH 5.5–5.7 with NH4OH) as mobile phase at 1 mL/min flow rate. Creatinine was monitored at 235 nm and the collected fraction was dried under vacuum at 40 °C. Creatinine was converted into its tert-butyldimethylsilyl derivative with 70 μL of CH3CN:MTBSTFA (2:1 by vol) at 70 °C for 30 min. Gas chromatographic separation was achieved with a 30-m SPB-35 column (Supelchem, Milan, Italy). The injector temperature was at 250 °C, the initial GC oven temperature was set at 170 °C for 1 min and subsequently increased to 180 °C at 2.5 °C/min, and to 270 °C at 30 °C/min. Injections of samples were alternated with duplicate analysis of calibrators having 1H:2H ratios of 0.8, 1.0, 1.2, 1.0, 0.8, etc. The isotopic ratio was determined by monitoring ions at m/z 298 and 301 for unlabeled and labeled creatinine, respectively.

Concentration of serum creatinine (μmol/L) was then computed from the measured isotopic ratio on the basis of the weight of each serum aliquot, the density, and the internal calibrator added, as already described (7).

experimental design

Eight different materials were sent to 51 clinical laboratories of the Lombardy region: two fresh-frozen human serum pools (CON1 and CON2) and six lyophilized materials (LYO1–LYO6). The frozen pools were delivered in solid CO2, stored at −20 °C, thawed on the day of analysis, and analyzed within 1 h. Lyophilized sera were stored at 4 °C and reconstituted 1 h before analysis. In each material, creatinine was measured in quintuplicate in three consecutive days with the automated analyzers routinely used (15 results per laboratory, per control material). The study participants were asked to classify their analytical method according to the chemical principle, the instrumentation, the source of reagents, and the type of calibrator. According to this classification we identified three homogeneous groups [Boehringer–Hitachi, Johnson & Johnson (J&J), Beckman] and two miscellaneous groups.

control materials

CON1 and CON2 were prepared from sera obtained with Serum Separator Tubes (SST Vacutainer; Becton Dickinson, Milan, Italy). Concentration was adjusted by adding appropriate amounts of creatinine (SRM 914a). LYO1–LYO6 were lyophilized commercial materials: LYO1 (Roche N, lot no. A 1136); LYO2 (Roche A, lot no. S 1135 2); LYO3 (Boehringer, Precinorm U, lot no. 177111 61); LYO4 (Boehringer, Precipath A, lot no. 177481 71); LYO5 (Bio-Rad, Lyphochek 1, lot no. 15011); and LYO6 (Bio-Rad, Lyphochek 2, lot no. 15012).

instrumentation

Analytical instruments used in this experiment were: Boehringer Hitachi analyzers 704 (2), 717 (7), 747 (6), 911 (3), (Boehringer Mannheim, Milan, Italy); Beckman CX7 (4), CX3 (1), CX5 (1) (Beckman Analytical, Cassina de Pecchi, Italy); Dax 24 (1) (Bayer, Cavenago, Italy); Olympus AU 5000 (3), Au 510 (1) (Kontron Instruments, Milan, Italy); Shimadzu CL 7000 (1), 7200 (1) (Shimadzu Italia, Milan, Italy); IL 900 (4), ILAB 1800 (2), Monarch (1) and Phoenix (1) (Instrumentation Laboratory, Milan, Italy); Ektachem analyzers 700 XR (8), 500 (3), 250 (1), (J&J, Cinisello Balsamo, Italy).

software

EQAS data were collected via an ad hoc computer program compiled in CA-Clipper Version 5.2 (Computer Associates, Milan, Italy) and distributed on floppy disk together with the samples. Data were automatically transferred in a Lotus 1-2-3 spreadsheet (release 3.1; Lotus Italia, Milan, Italy).

statistical analysis

The mean of each analytical run, the laboratory mean (mean of three analytical runs), the group mean, and the grand mean (mean of laboratory means) were calculated. SD and within-run CV (CVw), between-run CV (CVb, containing only the across-day component of variability), between-laboratories CV (CVinter), and overall CV (CVovr) were calculated with analysis of variance performed on a Lotus 1-2-3 spreadsheet.

Calibration bias line.

For each peer group we calculated the equation of the line defined by the two frozen pools (CON1 and CON2): where y is peer group mean and x is the IDMS value; fixed constant calibration bias (ap) and fixed proportional calibration bias (bp − 1) of each peer group were obtained from the parameters of the line.

Statistical verification of matrix effect occurrence.

Each laboratory mean, obtained for every lyophilized material, was corrected for the calibration bias of the laboratory itself according to the following formula: where YiL is the mean of lyophilized sera L of the laboratory i and ai, bi are parameters of the laboratory calibration bias line.

The statistical significance of the difference between corrected results, grouped according to the peer groups, and ID GC-MS value of each material was calculated (Student’s t-test). A statistically significant difference indicates the presence of a matrix bias (i.e., noncommutability of the material).

Matrix bias correction factor.

A factor to correct bias introduced by the matrix of the lyophilized control materials has been obtained by modifying the formula proposed by Ross et al. (6) to take into account the problem of the constant component of the calibration bias, very common in creatinine measurement with routine methods. The algorithm proposed by Ross et al. (6) for the calculation of the matrix bias correction factor of lyophilized sera is: where Yp F is the peer group mean of fresh frozen human pool.

Yp L is the peer group mean of lyophilized sera, CF is the GC-IDMS value of fresh frozen human pool, and CL is the GC-IDMS value of lyophilized sera.

Modified algorithm: where bp and ap are the parameters of the calibration bias line of a peer group method p.

Results

The reliability of our ID GC-MS method is demonstrated by the results obtained on NIST reference materials SRM 909a1: 84.3 μmol/L, CV 1.05% and 909a2: 470.0 μmol/L, CV 0.41%.

An overview of all results obtained on the six lyophilized materials and the two frozen pools is shown in Table 1⇓ . We report ID GC-MS target values and overall and peer group means. Results obtained by the clinical laboratories (including overall means and ANOVA) are also summarized. In some cases, such as LYO5, very large discrepancies among method means are evident.

The results obtained on the two frozen human serum pools were used to calculate constant calibration bias and proportional calibration bias of three homogeneous groups of analytical systems (we considered homogeneous the groups constituted by instruments, reagents, and calibrators from the same manufacturer). Table 2⇓ shows the biases from the ID GC-MS values and the parameters of the lines obtained.

The results of the statistical verification of the occurrence of matrix effect are presented in Table 3⇓ . Only three of 18 material/analytical system combinations exhibit commutable behavior.

By using the formula illustrated in Materials and Methods, one can calculate a “matrix bias correction factor” taking into account the different components of the calibration bias. Table 4⇓ shows the matrix bias correction factor of each lyophilized material for the different method groups. By multiplying the peer group means by these factors, one can remove the component of intermethod variability due to matrix effects from the results obtained on lyophilized materials. Fig. 1⇓ shows the peer group means obtained, for each material, before and after results modification according to the matrix bias correction factors. In Fig. 2⇓ comparability of data achievable on fresh frozen sera and lyophilized sera after correction is shown (J&J method group).

Figure 1.

Effects of the application of matrix bias correction factor.

Each bar represents the mean percent bias of every peer group mean from ID GC-MS value, before (a) and after (b) the application of the matrix bias correction factor. Plot (b) is representative of the calibration bias. Arrows indicate the frozen pools.

Figure 2.

Bias/concentration profile of J&J peer group.

Mean values before (open symbols) and after (closed symbols) correction are compared with mean values of fresh frozen sera (x-axis). (♦, ⋄, LYO5; ▪, □, LYO1; ▾, ▿, LYO3; ✚, ✙, LYO4; ▴, ▵, LYO2; ⬡, ⬡, LYO6; ×, CON1 and CON2).

Table 1.

Results of creatinine proficiency testing (μmol/L).

Table 2.

Data of calibration bias lines for three homogeneous groups of analytical systems.

Table 3.

Results (μmol/L) obtained on lyophilized materials corrected for calibration bias and significance of difference (μmol/L) between corrected values and definitive method values.

Table 4.

Matrix bias correction factors of each material for the different method groups.

Discussion

We developed an ID GC-MS method similar to the one proposed by Stöckl and Reinauer (8) that combines a sufficient practicability with good precision (CVs from 0.41% to 1.42%) (Table 1⇑ ) and accuracy (bias of 0.36% and 1.51% from NIST target values on 909a1 and 909a2, respectively).

Our experiments (see Table 1⇑ and Fig. 1a⇑ ) emphasize that: (a) there are very discordant percent biases from the ID GC-MS target values—very high for control materials with lower creatinine concentrations, very small for sera with higher concentrations; (b) the major component of variability is the between-laboratories variability that is always very high; (c) the intralaboratory variability can be considered acceptable but, especially at the lower concentrations, it is far from the analytical goal calculated on the basis of biological intraindividual variability (1); and (d) the frozen pools (CON1 and CON2), although showing very similar intralaboratory variability, exhibit a lower interlaboratory imprecision. The results are comparable (in terms of imprecision and inaccuracy) with those obtained in a previous experiment (9). Large differences among the method means were present (Table 1⇑ ). In particular, LYO5 shows bias of >50% between enzymatic and picrate methods, suggesting the presence of some noncreatinine substance reacting with picrate. Unfortunately, 16 laboratories were working with miscellaneous conditions [calibrators and (or) reagents from manufacturers different from those of the instrumentation] or with unique systems, and it was not possible to classify and treat those data. Also, the enzymatic group is not homogeneous, with one laboratory using the UV creatinine reaction and another using the Trinder coupled reaction. For these reasons we performed further calculations only for the three homogenous groups of analytical systems: J&J analyzers, Beckman CX family, and Boehringer–Hitachi family.

Assuming the frozen pools as not affected by any matrix effect, we used them to calculate calibration bias (e.g., method bias observed relative to ID GC-MS method) according to Ross et al. (6). Percent biases obtained on CON1 were completely different from those on CON2, thus suggesting the occurrence of a significant constant calibration bias (Tables 1⇑ and 2⇑ ). We decided to take into account constant calibration bias by calculating the equation of the line defined by the two pools. The data of slope and intercept (Table 2⇑ ) clearly individuate a different behavior of the three peer groups. Note the similarity of the parameters of our regression line for Hitachi systems with the equation of the correlation between an HPLC reference method and the Hitachi 911 results presented by Blijenberg et al. (5). Clearly the methods based on the Jaffe reaction are affected by an important positive constant calibration bias (Table 2⇑ ) caused probably by an aspecific signal. This is particularly evident at low creatinine concentrations or with some type of artificial material such as LYO5. This positive bias, in the case of the Boehringer–Hitachi group, can be almost completely attributed to the picrate reactivity with proteins. The reading window of the Boehringer method is quite long (∼90 s), with a prolonged delay from the starter addition (∼90 s). This favors the interference from slow-reacting interferents such as proteins (10). In fact, an extensively dialyzed albumin solution (50 g/L) gives (on an Hitachi 747) an apparent creatinine value of 21 ± 0.9 μmol/L. The apparent accuracy displayed for samples with intermediate concentration is due to a concomitant negative proportional bias. Better performances were obtained with enzymatic methods, both for dry and wet chemistry (Table 1⇑ ). In particular, laboratories using wet chemistry enzymatic methods provided very promising results. This finding is in agreement with Blijenberg et al. (3)(4), but the very limited number of participants using these methods (two) does not allow any generalization.

Table 3⇑ shows clearly that lyophilized sera behave differently from the frozen pools. Only in three of 18 material/method combinations was the difference between the two types of materials not significant. These results imply that the use of target values on these types of materials is useless and can lead to faulty considerations. The bias introduced by the matrix is typical for a defined analytical system. Fig. 1a⇑ shows how different this effect is for the various materials and analytical systems. With the application of the algorithm proposed, it is possible to calculate factors (shown in Table 4⇑ ) that are able to correct for the error introduced by the matrix. Fig. 1b⇑ , in which the matrix effect is corrected, shows almost identical behavior for the different materials with similar creatinine content, whether frozen or lyophilized. Indeed the bias/concentration profile of results obtained on lyophilized sera after correction closely resembles behavior of fresh frozen sera (Fig. 2⇑ ). The proposed algorithm has a more general applicability than the previous one (6) and can give reliable results even when a constant calibration bias is present.

All matrix bias correction factors were calculated with the peer group means, but we tried also to calculate the factors by using single laboratory data of the same peer group. The results obtained showed a noteworthy concordance among laboratories of the same group. The variability of the obtained factors, measured as CV, ranged between 0.80% and 3.75% according to the material and the group of methods. This homogeneity of data allows us to hypothesize the possibility of the use of a relatively small number of pilot laboratories to calculate the matrix bias correction factor for a defined lot of control material to be used in an EQAS.

The major problem of EQAS, when artificially manipulated control materials are involved, is the bias introduced by the materials themselves for the different types of methods. This fact forces the use of peer group means, but without any guarantee, apart from the producer declaration, of the real accuracy of the analytical system. However, it is not possible to verify whether the difference among the various analytical systems are caused by the characteristics of the material only or by real accuracy problems with a risk “of an implicit endorsement of methodologies that fail to satisfy fundamental accuracy goals” (11). Obviously the more straightforward approach to this problem should be the use of fully commutable material such as fresh or frozen sera, but the costs of distributing this type of material prevent its use, at least on a regular basis. The matrix-adjusted target values can be an acceptable compromise that allows the utilization of the lyophilized sera provided that two important limitations are adequately considered: (a) the matrix bias correction factor can be calculated only for well-defined analytical systems; (b) the serum pools used in generating the algebraic correction are the same as normal fresh serum specimens. The last one can be an important drawback; the probability that a minimally manipulated serum pool could exhibit a noncommutable behavior is low, but a check of the commutability, e.g., according to the College of American Pathologists’ protocol (12), is advisable. Moreover, this approach is not intended to substitute the direct comparison with a Reference Method on fresh sera (13), but only to minimize the matrix effect, thus allowing the use of Reference Method target values for lyophilized materials.

Acknowledgments

This work was supported by grant no. 869, funded by Lombardy Region, Italy. The assays performed in this study were made possible through the efforts of: L. Guerrini, M. Cavalleri, C. Petrini, A. Bianchi Bosisio, E. Solbiati, P. Cueroni, A. Nespolo, E. Scarazzatti, R. Vigoni, S. Bossini, A. Petrella, P. Maestrini, F. Tirelli, A. Cespa, L. Ferrari, G. Casiraghi, C. Okely, R. Antinozzi, F. Mariani, M.L. Carati, A. Ferrari, T. Baratto, G. De Leo, C. Ottomano, E. Guagnellini, G. Gallina, G. Giocoli, M. Musmeci, V. Malacrida, M. Iannone, A. Marocchi, M. Panteghini, M. Mancosu, A. Marelli, R. Trotti, F. Aguzzi, P. Mocarelli, A. Pagano, P.A. Bonini, and M. Murone.

Footnotes

  • ↵1 Nonstandard abbreviations: EQAS, external quality-assessment scheme; ID GC-MS, isotope dilution gas chromatography–mass spectrometry; SRM, Standard Reference Material; and MTBSTFA, N-methyl-N-(tert-butyldimethylsilyl)-trifluoroacetamide.

  • 1P <0.05; NS, not significant.

  • © 1997 The American Association for Clinical Chemistry

References

0 Thoughts to “Bossini 4 Assignments

Leave a comment

L'indirizzo email non verrà pubblicato. I campi obbligatori sono contrassegnati *