PatentDe  


Dokumentenidentifikation EP0469885 18.12.1997
EP-Veröffentlichungsnummer 0469885
Titel Gewichtetes Summierungsleistungssystem
Anmelder General Electric Co., Schenectady, N.Y., US
Erfinder Engeler, William Ernest, Scotia, New York 12302, GB
Vertreter Voigt, R., Dipl.-Ing., Pat.-Anw., 65812 Bad Soden
DE-Aktenzeichen 69128174
Vertragsstaaten DE, FR, GB, IT, NL
Sprache des Dokument En
EP-Anmeldetag 31.07.1991
EP-Aktenzeichen 913070132
EP-Offenlegungsdatum 05.02.1992
EP date of grant 12.11.1997
Veröffentlichungstag im Patentblatt 18.12.1997
IPC-Hauptklasse G06F 15/80
IPC-Nebenklasse G06G 7/60   

Beschreibung[en]

The invention relates to analog computer structures for performing a plurality of weighted summations of a plurality of input signals (as are used in neural nets, for example) and more particularly, to adapting such analog computer structures for use with digital electronic circuits.

Computers of the von Neumann type architecture have limited computational speed owing to the communication limitations of the single processor. These limitations can be overcome if a plurality of processors are utilized in the calculation and are operated at least partly in parallel. This alternative architecture, however, generally leads to difficulties associated with programming complexity. Therefore, it is often not a good solution. Recently, an entirely different alternative that does not require programming has shown promise. The networking ability of the neurons in the brain has served as a model for the formation of a highly interconnected set of analog processors, called a "neural network" or "neural net" that can provide computational and reasoning functions without the need of formal programming. The neural nets can learn the correct procedure by experience rather than being preprogrammed for performing the correct procedure. The reader is referred to R.P. Lippmann's article "An Introduction to Computing With Neural Nets" appearing on pages 4-21 of the April 1987 IEEE ASSP MAGAZINE (0740-7467/87/0400-0004/$10.00 " 1987 IEEE), incorporated herein by reference, for background concerning neural nets,

Neural nets are composed of a plurality of neuron models, analog processors each exhibiting "axon" output signal response to a plurality of "synapse" input signals. In a type of neural net called a "perceptron", each of these processors calculates the weighted sum of its "synapse" input signals, which are respectively weighted by respective weighting values that may be positive- or negative-valued, and responds non-linearly to the weighted sum to generate the "axon" output response. This relationship may be described in mathematical symbols as follows.

Here, i indexes the input signals of the perceptron, of which there are an integral number M, and j indexes its output signals, of which there are an integral number N. Wi,j is the weighting of the ith input signal as makes up the jth output signal at such low input signal levels that the function

is approximately linear. At higher absolute values of its argument, the function

no longer exhibits linearity but rather exhibits a reduced response to

This type of non-linear response is termed "sigmoidal". The weighted summation of a large number of sampled-data terms can be viewed as the process of correlating the sampled-data function described by those sampled-data terms with the sampled-data function described by the pattern of weights; and the analog processor used as a neuron model in a "perceptron" can be viewed as a correlator with non-linear circuitry exhibiting a sigmoidal response connected thereafter.

A more complex artificial neural network arranges a plurality of perceptrons in hierarchic layers, the output signals of each earlier layer providing input signals for the next succeeding layer. Those layers preceding the output layer providing the ultimate output signal(s) are called "hidden" layers. The training of an artificial neural network comprising hierarchic layers of perceptrons is accomplished in the prior art by using a back-propagation training algorithm.

The back-propagation training algorithm is an iterative gradient algorithm designed to minimize the mean square error between the actual output of a multi-layer feed-forward neural net and the desired output. It requires continuous, differentiable non-linearities in the transfer function of the non-linear circutry used after the weighted summation circuits in the neural net layer. A recursive algorithm starting at the output nodes and working back to the first hidden layer is used iteratively to adjust weights in accordance with the following formula. Wi,j(t+1) = Wi,j(t) - ηδj xi In this equation Wi,j(t) is the weight from hidden node i (or, in the case of the first hidden layer, from an input node) to node j at time t; xi is either the output of node i (or, in the case of the first hidden layer, is an input signal); η is a gain term introduced to maintain stability in the feedback procedure used to minimize the mean square errors between the actual output(s) of the perceptron and its desired output(s); and δj is a derivative of error. The general definition of δj is the change in error energy from output node j of a neural net layer with a change in the weighted summation of the input signals used to supply that output node j.

Lippman presumes that a particular sigmoid logistic non-linearity is used. Presuming the non-linearity of processor response is to be defined not as restrictively as Lippmann does, then δj can be more particularly defined as in equation (2), following, if node j is an output node, or as in equation (3), following, if node j is an internal hidden node. δj = zj&min; (dj - zj)

In equation (3) δj and zj are the desired and actual values of output response from the output layer and zj&min; is differential response of zj to the non-linearity in the output layer - i.e., the slope of the transfer function of that non-linearity. In equation (4) k is over all nodes in the neural net layer succeeding the hidden node j under consideration and Wj,k is the weight between node j and each such node k. The term zj&min; is defined in the same way as in equation (3).

The general definition of the zj&min; term appearing in equations (3) and (4), rather than that general term being replaced by the specific value of zj&min; associated with a sigmoid logistic non-linearity, is the primary difference between the training algorithm as described here and as described by Lippmann. Also, Lippmann defines dj in opposite polarity from equations (1), (3) and (4) above.

During training of the neural net, prescribed patterns of input signals are sequentially repetitively applied, for which patterns of input signals there are corresponding prescribed patterns of output signals known. The pattern of output signals generated by the neural net, responsive to each prescribed pattern of input signals, is compared to the prescribed pattern of output signals to develop error signals, which are used to adjust the weights per equation (2) as the pattern of input signals is repeated several times, or until the error signals are detected as being negibly valued. Then training is done with the next set of patterns in the sequence. During extensive training the sequence of patterns may be recycled.

In the present-day development of the integrated electronic circuitry art, the weighted summation of a large number of terms, each of which has resolution that would require plural-bit digital sampling, can be done appreciably faster and at less cost in integrated circuit die area by processing in the analog regime rather than in the digital regime. Using capacitors to perform weighted summation in accordance with Coulomb's Law provides neural nets of given size operating at given speed that consume less power than those the analog processors of which use resistors to implement weighted summation in accordance with Ohm's Law.

There is a class of problems that can usefully employ neural nets for their solution, but for which synapse input signal samples are plural-bit digital samples. Particularly where there are numerous plural-bit digital synapse input signal samples, difficulties are encountered in getting the plural-bit digital samples into the monolithic integrated circuit in which a neural net layer reposes, because of practical limitations on the number of pins available on the integrated circuit package. An example of when this type of problem is encountered is when one attempts to use a neural net to perform the two-dimensional spatial filtering associated with the recognition of shapes in video signals generated by raster scanning a focal plane array of photosensors in a video camera. In such two-dimensional spatial filtering it is desirable to provide for the temporary storage of the raster-scanned video signal samples, so that they may be temporally aligned for application to the neural net. In the present state of technology the temporary storage of a substantial number of video samples is most advantageously done in digital memory. The synapse input signals made available to the neural net from the digital memory are in digital, not analog, form.

The problem of the synapse input signals made available to a neural net being in plural-bit digital form rather than in analog form can arise in other contexts, too. In systems where the neural net layers are not located in proximity to each other the axon output response of a hidden neural net layer can be digitized in an analog-to-digital converter and transferred without error over substantial distances to another neural net layer. Using digital rather than analog signals to communicate between neural net layers can also be useful when back-propagation training algorithms are used to train a neural net, since it easier in the digital regime to alter the training algorithm in specified ways to take into account whether a new task is being learned or whether instead an old task is being modified to take into account a change in the operating environment.

Streams of digital samples have to be converted to respective analog input signals in order to be used with the neural nets known in the prior art; and, since the samples requiring conversion are parallel in time, a respective digital-to-analog converter is required for each input signal to the neural net. Except where the streams of digital samples are single-bit in nature, the well-known problems of avoiding the conversion non-linearity associated with converting a multiple-bit digital number code stream to a respective analog signal have to be faced with regard to each of these digital-to-analog converters, and these problems are known to be more difficult to solve as the size of the converters has to be made smaller to accommodate a large number of them on a monolithic integrated circuit.

Digital signals for application to an artificial neural network can be processed on a bit-sliced basis, so there is no need for digital-to-analog converters to convert multiple-bit digital number code streams each to a respective analog signal. A system for performing N weighted summations of input signals, M in number, identified by respective ones of consecutive ordinal numbers first through Mth, N being an integer greater than zero and M being an integer greater than one, that embodies the invention is characterized in that each of the input signals is a digital signal and that the system includes apparatus for performing said N weighted summations on a bit-slice basis. The weighted summation procedure when there is more than one bit-slice in the digital input signals comprises: (a) the generation of a respective partial weighted summation for each bit slice and (b) a final weighted summation, in which the partial weighted summation results are weighted in accordance with the significance of the bit slices with which they are respectively associated and are after such weighting summed together. The final weighted summation of the partial weighted summation results, which are analog in nature, can be carried out in the analog regime (and thereafter digitized, if so desired). Alternatively, the partial weighted summation results may be digitized and then summed with appropriate shifts to obtain final weighted summation results in digital form.

The various figures of the drawing will be described briefly, before considering them in greater detail.

FIGURE 1 is a schematic diagram of a prior-art single-ended charge sensing amplifier comprising a Miller integrator with resetting circuitry that compensates for input offset error in the differential-input operational amplifier the Miller feedback capacitor provides degenerative feedback to; this type of single-ended charge sensing amplifier and a balanced version thereof are preferred over simpler Miller integrators for the charge sensing amplifiers used in the circuitry for performing weighted summations in the analog regime.

FIGURES 2, 3 and 4 are schematic diagrams of three different apparatuses, each for performing a plurality of weighted summation procedures in parallel on a bit slice of input signals.

FIGURE 5 is a schematic diagram of apparatus for performing a plurality of weighted accumulation procedures, each of which weighted accumulation procedures weights and sums sequentially supplied data.

FIGURE 6 is a schematic diagram of a neural net layer including any of the FIGURE 2, FIGURE 3 and FIGURE 4 apparatuses for performing a plurality of weighted summation procedures; possibly a FIGURE 5 apparatus for performing a plurality of weighted accumulation procedures; an array of non-linear processors and an array of analog-to-digital converters.

FIGURE 7 is a schematic diagram of a voltage amplifier that can be used as a non-linear processor in neural net layers per FIGURE 6.

FIGURE 8 is a schematic diagram of apparatus for digitizing analog partial weighted summation results as sequentially supplied from one of the FIGURE 2, FIGURE 3 and FIGURE 4 apparatuses and then digitally performing a plurality of weighted accumulation procedures, each of which weighted accumulation procedures weights and sums a respective succession of digitized sequentially supplied data.

FIGURES 9A and 9B, arranged respectively on the left and on the right of the other, together form a schematic diagram (generally referred to as FIGURE 9) of a neural net layer including any of the FIGURE 2, FIGURE 3 and FIGURE 4 apparatuses for performing a plurality of partial weighted summation procedures; an array of analog-to-digital converters for digitizing the partial weighted summation results; possibly a FIGURE 8 apparatus for digitally performing a plurality of digital final weighted summation procedures; and an array of digital non-linear processors generating digital axonal responses to the digital final weighted summation results.

FIGURE 10 is a schematic diagram of a digital non-linear processor that can be used in neural net layers.

FIGURE 11 is a graph showing the non-linear transfer function of the FIGURE 10 digital non-linear processor.

FIGURES 12 and 13 are schematic diagrams of two further apparatuses, each for performing a plurality of weighted summation procedures in parallel on a bit slice of input signals.

FIGURE 14 is a schematic diagram of analog apparatus for performing a plurality of weighted accumulation procedures, each of which weighted accumulation procedures weights and sums sequentially supplied data.

FIGURE 15 is a schematic diagram of a neural net layer including either of the FIGURE 12 and FIGURE 13 apparatuses for performing a plurality of weighted summation procedures; possibly a FIGURE 14 apparatus for performing a plurality of weighted accumulation procedures; an array of non-linear processors and an array of analog-to-digital converters.

FIGURES 16 and 17 are schematic diagrams of respective modifications of the FIGURE 15 neural net layer to adapt it for operation with one's complement synapse input signals rather than with two's complement synapse input signals.

FIGURE 18 is a schematic diagram of apparatus for digitizing analog partial weighted summation results as sequentially supplied from one of the FIGURE 12 and FIGURE 13 apparatuses and then digitally performing a plurality of weighted accumulation procedures, each of which weighted summation procedures weights and sums a respective succession of digitized sequentially supplied data.

FIGURES 19A and 19B, arranged respectively on the left and on the right of the other, together form a schematic diagram (generally referred to as FIGURE 19) of a neural net layer including two ranks of either of the FIGURE 12 and FIGURE 13 apparatuses for performing a plurality of partial weighted summation procedures; two ranks of analog-to-digital converters for digitizing the partial weighted summation results; two ranks of the FIGURE 18 apparatus connected for digitally performing a plurality of digital final weighted summation procedures; and an array of digital non-linear processors generating digital axonal responses to the digital final weighted summation results.

FIGURES 20A and 20B, arranged respectively on the left and on the right of the other, together form a FIGURE 20 which is a schematic diagram of a modification that can be made manifold times to the neural net shown in FIGURES 2, 8 and 9; FIGURES 3, 8 and 9; FIGURES 12, 18 and 19; or FIGURES 13, 18 and 19 for the programmable weighting of the capacitances used in performing weighted summations of synapse signals.

FIGURES 21A and 21B, arranged respectively on the left and on the right of the other, together form a FIGURE 21 which is a schematic diagram of a modification that can be made manifold times to the neural net shown in FIGURES 4, 8 and 9, for the programmable weighting of the capacitances used in performing weighted summations of synapse signals.

FIGURES 22A and 22B, arranged respectively on the left and on the right of the other, together form a FIGURE 22 which is a schematic diagram of a modification that can be made manifold times to the neural net shown in FIGURES 12, 18 and 19 or in FIGURES 13, 18 and 19, for the programmable weighting of the capacitances used in performing weighted summations of synapse signals.

FIGURES 23A and 23B, arranged respectively on the left and on the right of the other, together form a FIGURE 23 that is a schematic diagram of a neural net layer in which both forward propagation and back propagation through a capacitive network are carried out with balanced signals, in which digital capacitors have their capacitance values programmed from respective word storage locations within a digital memory, and in which the signs of the digital signals being weighted and summed are taken into account in the final weighted summation procedure performed on a digital basis.

FIGURES 24A and 24B, arranged respectively on the left and on the right of the other, together form a FIGURE 24 that is a schematic diagram of an alternative neural net layer in which both forward propagation and back propagation through a capacitive network are carried out with balanced signals, and in which digital capacitors have their capacitance values programmed from respective word storage locations within a digital memory. In the FIGURE 24 neural net layer the signs of the digital signals being weighted and summed are taken into account in the partial weighted summation procedure performed on an analog basis.

FIGURE 25 is a schematic diagram of a back-propagation processor used for training a neural net having layers as shown in FIGURES 2, 8 and 9 or in FIGURES 3, 8 and 9 modified manifoldly per FIGURE 20; or a neural net having layers as shown in FIGURES 4, 8 and 9 modified manifoldly per FIGURE 21; or a neural net having layers as shown in FIGURES 12, 14 and 15 or in FIGURES 13, 14 and 15 modified manifoldly per FIGURE 22; or a neural net having layers as shown in FIGURE 23; or or a neural net having layers as shown in FIGURE 24.

FIGURE 26 is a schematic diagram of a system having a plurality of neural net layers as shown in FIGURES 2, 8 and 9 or in FIGURES 3, 8 and 9 modified manifoldly per FIGURE 20; or in FIGURES 4, 5 and 6 modified manifoldly per FIGURE 21; or in FIGURES 12, 14 and 15 or in FIGURES 13, 14 and 15 modified manifoldly per FIGURE 22; or in FIGURE 23; or in FIGURE 24 each of which neural net layers has a respective back-propagation processor per FIGURE 23 associated therewith.

FIGURES 27A and 27B, arranged respectively on the left and on the right of the other, together form a FIGURE 27 that is a schematic diagram of an alternative modification of that can be made manifold times to the neural net shown in FIGURES 2, 8 and 9 or in FIGURES 3, 8 and 9, for the programmable weighting of the capacitances used in performing weighted summations of synapse signals.

FIGURES 28A and 28B, arranged respectively on the left and on the right of the other, together form a FIGURE 28 that is a schematic diagram of an alternative modification of that can be made manifold times to the neural net shown in FIGURES 4, 8 and 9, for the programmable weighting of the capacitances used in performing weighted summations of synapse signals.

FIGURES 29A and 29B, arranged respectively on the left and on the right of the other, together form a FIGURE 29 that is a schematic diagram of an alternative modification of that can be made manifold times to the neural net shown in FIGURES 12, 14 and 15 or in FIGURES 13, 14 and 15, for the programmable weighting of the capacitances used in performing weighted summations of synapse signals.

FIGURES 30A and 30B, arranged respectively on the left and on the right of the other, together form a FIGURE 30 that is a schematic diagram of a neural net layer in which both forward propagation and back propagation through a capacitive network are carried out with balanced signals, in which digital capacitors have their capacitance values programmed from counters, and in which the signs of the digital signals being weighted and summed are taken into account in the final weighted summation procedure performed on a digital basis.

FIGURES 31A and 31B, arranged respectively on the left and on the right of the other, together form a FIGURE 31 that is a schematic diagram of an alternative neural net layer in which both forward propagation and back propagation through a capacitive network are carried out with balanced signals, and in which digital capacitors have their capacitance values programmed from counters. In the FIGURE 31 neural net layer the signs of the digital signals being weighted and summed are taken into account in the partial weighted summation procedure performed on an analog basis.

FIGURE 32 is a schematic diagram of the arrangement of stages in the counters shown in FIGURES 27-31.

FIGURE 33 is a schematic diagram of the logic elements included in each counter stage in the counters shown in FIGURES 27-32.

More particularly, FIGURE 1 shows a single-ended charge sensing amplifier QSk of a preferable type for implementing the single-ended charge sensing amplifiers used in modifications of the FIGURE 2, 3, 12, 14, 20B, 21A, 22B, 27 and 29 B circuitry. The charge sensing amplifier QSk is essentially a Miller integrator that includes a differential-input operational amplifier OA having a Miller feedback capacitor MC connecting from its output connection OUT back to its inverting (-) input connection during normal operation. The non-inverting (+) input connection of the operational amplifier OA is connected to a fixed potential in the single-ended charge sensing amplifier QSk , which fixed potential is shown as having a value (VSS + VDD)/2 midway between a relatively high potential (VDD) and a relatively low potential (VSS). During normal charge-sensing operation a relatively low potential (VSS) is applied via RESET terminal to a logic inverter INV that responds to apply a relatively high potential (VDD) to a transmission gate TG1. The transmission gate TG1 is rendered conductive to connect the output connection OUT of operational amplifier OA to capacitor MC to complete its connection as Miller feedback capacitor. The relatively low potential applied via RESET terminal conditions a transmission gate TG2 and a transmission gate TG3 both to be non-conductive. QSK is a preferred charge sensing amplifier because differential input offset error in the operational amplifier OA is compensated against, owing to the way the Miller integrator is reset.

During periodic reset intervals for the integrator a relatively high potential (VDD) is applied via RESET terminal to condition transmission gates TG2 and TG3 each to be conductive and to condition the logic inverter INV output potential to go low, which renders transmission gate TG1 non-conductive. The conduction of transmission gate TG2 connects the output connection OUT of operational amplifier OA directly to its inverting (-) input connection, completing a feedback connection that forces the inverting (-) input connection to the differential input offset error voltage, which voltage by reason of transmission gate TG3 being conductive is stored on the Miller capacitor MC. When normal charge-sensing operation is restored by RESET terminal going low, the differential input offset error bias remains stored on the Miller capacitor MC, compensating against its effect upon charge sensing.

Supposing the operational amplifier OA to be a differential output type having balanced output connections OUT and OUTBAR, a balanced version of the charge sensing amplifier QSK can be formed by disconnecting the non-inverting (+) input connection of the operational amplifier OA from a point of fixed potential having a value (VSS + VDD)/2. Instead, the non-inverting (+) input connection of the operational amplifier OA is arranged to have a feedback connection from the OUTBAR output connection of the operational amplifier OA similar to the feedback connection from the OUT output connection of the operational amplifier OA to its inverting (-) input connection. This balanced version of the charge sensing amplifier QSK is shown throughout the remaining figures of the drawing.

M is a positive plural integer indicating the number of input signals to the FIGURE 2, 3, 4, 12 or 13 weighted summation apparatus, and N is a positive plural integer indicating the number of output signals the FIGURE 2, 3, 4, 12 or 13 apparatus can generate. To reduce the written material required to describe operation of the weighted summation apparatuses in FIGURES 2, 3, 4, 12 or 13 of the drawing, operations using replicated elements will be described in general terms; using a subscript i ranging over all values one through M for describing operations and circuit elements as they relate to the (column) input signals x&sub1;, x&sub2;,... xM; and using a subscript j ranging over all values one through N for describing operations and apparatus as they relate to the (row) output signals y&sub1;, y&sub2;,...yN. That is, i and j are the column and row numbers used to describe particular portions of the FIGURES 2, 3, 4, 12 and 13 weighted summation apparatuses and modifications of those apparatuses.

The FIGURE 2 apparatus performs a plurality of weighted summation procedures in parallel on each successive bit slice of input signals, which input signals comprise a plurality M in number of parallel bit streams x&sub1;, x&sub2;,... xM. This apparatus is assumed to receive a first operating voltage VDD, a second operating voltage VSS, and a third operating voltage (VSS + VDD)/2 midway between VSS and VDD. VDD and VSS are presumed to be relatively positive and relatively negative respective to each other.

Each input voltage signal xi is applied as control signal to a respective multiplexer MXi and to a respective multiplexer MX(i+M) . Multiplexer MXi responds to xi being a ONE to apply the VDD first operating voltage to an input line ILi and responds to xi being a ZERO to apply the VSS second operating voltage to the input line ILi. Multiplexer MX(i+M) responds to xi being a ONE to apply the VSS second operating voltage to an input line IL(i+M) and responds to xi being a ZERO to apply the VDD first operating voltage to the input line IL(i+M).

A capacitor Ci,j has a first plate connected to the input line ILi and has a second plate connected to an output line OLj. A capacitor C(i+M),j has a first plate connected to the input line IL(i+M) and has a second plate connected to the output line OLj. Capacitor Ci,j and capacitor C(i+M),j together are considered as a pair, for providing in effect a single weighting capacitor, the capacitance of which is the difference in the capacitances of capacitor Ci,j and capacitor C(i+M),j between their respective first and second plates. The charge placed on the output line OLj by all the weighting capacitors connecting thereto is sensed on a single-ended basis by a respective charge-sensing amplifier QSj. Each of the charge-sensing amplifiers QSj is shown as a respective Miller integrator comprising an operational amplifier and Miller feedback capacitors.

During reset or zeroing of all the charge-sensing amplifiers QSj, each of the xi input voltages is a logic ZERO. This applies VSS to the plates of capacitors Ci,j connected from the multiplexers MXi and applies VDD to the plates of capacitors C(i+M),j connected from the multiplexers MX(i+M). The total capacitance on each output line OLj is caused to be the same as on each of the other output lines by a respective shunt capacitor C0,j to signal ground, which capacitor either has capacitance that is so large as to overwhelm the other capacitances on the output line OLj or preferably that complements the other capacitances on the output line OLj. Causing the total capacitance on each output line OLj to be the same as on each of the other output lines makes the sensitivities of the charge-sensing amplifiers QSj to their respective inputs uniform, presuming them to be Miller integrators of identical design. If the capacitances of capacitor Ci,j and capacitor C)i+M),j between their respective first and second plates sum to a prescribed standard value, for the complete selection range of i and j, the sensitivities of the charge-sensing amplifiers QSj to their respective inputs are uniform withou need for a respective shunt capacitor C0,j to signal ground for each output line OLj, presuming the charge-sensing amplifiers QSj to be Miller integrators of identical design.

After reset or zeroing, when xi bits for different i may each be ZERO or ONE, each xi bit that is a ZERO creates no change in charge condition on any of the output lines OLj. A bit xi that is a ONE creates an incremental change in charge on an output line OLj that, in accordance with Coulomb's Law, is equal to (VSS - VDD)/2 times the difference in the capacitances of capacitors Ci,j and C(i+M),j between their respective first and second plates. The sum of these incremental charges accumulates on the output line OLj and is sensed by the charge-sensing amplifier QSj.

FIGURE 3 shows a modification of the FIGURE 2 apparatus in which the multiplexer MXi and the multiplexer MX(i+M) respond to xi being a ZERO to apply the third operating voltage (VSS + VDD)/2 to the input line ILi and to the input line IL(i+M), respectively. During reset, the weighting capacitors Ci,j and C(i+M) will be charged to relatively small bias voltages between their plates, rather than to bias voltages of amplitudes close to (VSS + VDD)/2. The FIGURE 2 apparatus is advantageous over the FIGURE 3 apparatus in that, in the FIGURE 2 apparatus, accuracy of the the third operating voltage (VSS + VDD)/2 being exactly midway between the first operating voltage VDD and the second operating voltage VSS is not necessary for accuracy of the partial weighted summation results.

The FIGURE 4 apparatus also performs a plurality of weighted summation procedures in parallel on each successive bit slice of input signals, which input signals comprise a plurality M in number of parallel bit streams x&sub1;, x&sub2;, x&sub3;,... xM. Logic inverters INV&sub1;, INV&sub2;, INV&sub3;,...INVM respond to the current bits x&sub1;, x&sub2;, x&sub3;,... xM respectively with their respective logic complements. (The current bits x&sub1;, x&sub2;, X&sub3;, ... XM are assumed to be supplied in accordance with the positive logic convention.) The FIGURE 4 apparatus also is assumed to receive a relatively positive first operating voltage VDD, a relatively negative second operating voltage VSS, and a third operating voltage (VSS + VDD)/2 midway between VSS and VDD.

The logic inverter INVi responds to x&sub1; being a ZERO to apply VDD to an input line ILi and responds to x&sub1; being a ONE to apply VSS to the input line ILi. As in FIGURES 2 and 3, the charge-sensing amplifier QSj is one of a plurality, N in number, of identical charge-sensing amplifiers for sensing the difference in charges accumulated on a respective pair of output lines. In FIGURE 4 the charge-sensing amplifier QSj is arranged for differentially sensing charge and is connected to sense the difference in charges accumulated on output lines OLj and OL(j+N). The output lines OLj and OL(j+N) are charged from each input line input line ILi via a capacitor Ci,j and via a capacitor Ci,(j+N), respectively. Capacitor Ci,j and capacitor Ci,(j+N) together are considered as a pair, for providing in effect a single weighting capacitor, the capacitance of which is the difference in the capacitances of capacitor Ci,j and capacitor Ci,(j+N) between their respective first and second plates. The total capacitance on each output line OLj is maintained the same as on each of the other output lines by a respective shunt capacitor C0,j to signal ground; and the total capacitance on each output line OL(j+N) is maintained the same as on each of the other output lines by a respective shunt capacitor C0,(j+N) to signal ground.

Where the capacitances of capacitor Ci,j and capacitor Ci,(j+N) between their respective first and second plates are to be alterable responsive to digital programming signals, it is preferred that the capacitances of capacitor Ci,j and capacitor Ci,(j+N) between their respective first and second plates sum to a prescribed standard value for the complete selection range of i and j. It is further preferred that each pair of capacitors Ci,j and Ci,(j+N) have a corresponding further pair of capacitors Ci,(j+2N) and Ci,(j+3N) associated therewith, capacitor Ci,(j+2N) having a capacitance equal to that of capacitor Ci,(j+N) and connecting output line OLj to a point of connection Pi,j, and capacitor Ci,(j+3N) having a capacitance equal to that of capacitor Ci,j and connecting output line OL(j+N) to the same point of connection Pi,j. If all the points of connection Pi,j connect to signal ground, the capacitors Ci,(j+2N) for all values of i together provide for each value of j the respective shunt capacitor C0,j to signal ground, and the capacitors Ci,(j+3N) for all values of i together provide for each value of j the respective shunt capacitor C0,(j+N) to signal ground. This is taught in greater detail in U. S. patent application serial No. 526,470.

The FIGURE 4 apparatus may be modified to replace logic inverters INV&sub1;, INV&sub2;, INV&sub3;, ... INVM with non-inverting driver amplifiers. In such case the other output connections of the differential-output operational amplifiers in the charge sensing amplifiers QS&sub1;, QS&sub2;, QS&sub3;, ... QSN are used to supply the y&sub1;, y&sub2;, y&sub3;, ... yN partial weighted summations.

The FIGURE 4 apparatus may alternatively'be modified to augment logic inverters INV&sub1;, INV&sub2;, INV&sub3;, ... INVM with non-inverting driver amplifiers DA&sub1;, DA&sub2;, DA&sub3;, ... DAM respectively and to use each non-inverting driver amplifier DAi to drive the points of connection Pi,j for all values of j. This provides for full-capacitor bridge drives to each charge-sensing amplifier QSj, rather than half-capacitor bridge drives. The advantage of doing this is that the common-mode voltage on the output lines OLj and OL(j+N) is zero, so one does not have to rely as much on the common-mode suppression of the charge-sensing amplifier QSj to keep the integration of charge within the operating supply range of that amplifier.

Where the input signals to the FIGURE 2, FIGURE 3 or FIGURE 4 weighted summation apparatus comprise single-bit words, the N signals y&sub1;, y&sub2;, y&sub3;,...yN can, per FIGURE 6, be applied to an array of respective non-linear processors NLP&sub1;, NLP&sub2;, NLP&sub3;, . NLPN to generate a plurality, N in number, of analog axonal output responses. As shown in FIGURE 6, these analog axonal output responses can be digitized in respective analog-to-digital converters ADC&sub1;, ADC&sub2;, ADC&sub3;,...ADCN, for application to the succeeding circuitry NEXT. For example, this succeeding circuitry may be an output neural net layer where the neural net layer thusfar described is a hidden neural net layer. Oversampling analog-to-digital converters of the sigma-delta type are preferred for analog-to-digital converters ADC&sub1;, ADC&sub2;, ADC&sub3;,...ADCN when passing digitized axonal output responses to another weighted summation apparatus of one of the types illustrated in FIGURES 2-4, since the sigma-delta type of converter is inherently adapted to supplying its digitized response in bit-serial form.

FIGURE 5 shows further weighted summation apparatus that can be used with the weighted summation apparatus of FIGURE 2 or 3 to provide for the weighted summation of digital signals having plural-bit words. The plurality B in number of bits per digital word is assumed in FIGURE 5 to be four bits per digital word. The digital signals are placed in bit-serial form and in word alignment for application to the weighted summation apparatus of FIGURE 2 or 3, which processes the bit-serial digital signals on a bit-slice basis to generate B partial weighted summation results, one per bit slice, during each successive word interval. A SIGN BIT FLAG signal is provided to signal the parallel occurrence of the sign bits in each successive group of M input signal words. The SIGN BIT FLAG signal is used to synchronize the operation of a sequencer SEQ in the further weighted summation apparatus of FIGURE 5. (Where the synapse input signals x&sub1;, x&sub2;,... xM are unsigned, the SIGN BIT FLAG signal is replaced by a MOST SIGNIFICANT BIT FLAG signal, of course.) The FIGURE 5 weighted summation apparatus can be described as weighted accumulation apparatus since it accumulates the B partial weighted summation results supplied to it from the weighted summation apparatus of FIGURE 2 or 3, the accumulated partial weighted summation result term being weighted in accordance with the significance of the bit slice involved in its generation.

The FIGURE 5 apparatus includes a plurality, N in number, of weighted summers generating respective ones of N final weighted summation results z&sub1;, z&sub2;, ..zN Each of these weighted summers weights the four partial weighted summation results yj sequentially supplied during a word interval as part of one of the output signals of the weighted summation apparatus of FIGURE 2 or 3, the weighting being done in accordance with the significance of the bits used to generate each respective partial weighted summation result, and sums the weighted partial weighted summation results to generate a final weighted summation result zj.

More particularly, when the sequencer SEQ determines that the least significant of the four partial weighted summation results yj associated with a digital word interval is being generated by the weighted summation apparatus of FIGURE 2 or 3, the sequencer SEQ directs sample-and-hold circuits SH1j and SH5j to sample and hold that yj value. When the sequencer SEQ determines that the next to least significant of the four partial weighted summation results yj associated with a digital word interval is being generated by the weighted summation apparatus of FIGURE 2 or 3, the sequencer SEQ directs sample-and-hold circuits SH2j and SH6j to sample and hold that yj value. When the sequencer SEQ determines that the next to most significant of the four partial weighted summation results yj associated with a digital word interval is being generated by the weighted summation apparatus of FIGURE 2 or 3, the sequencer SEQ directs sample-and-hold circuits SH3j and SH7j to sample and hold that yj value. When the sequencer SEQ determines that the most significant of the four partial weighted summation results yj associated with a digital word interval is being generated by the weighted summation apparatus of FIGURE 2 or 3, the sequencer SEQ directs a sample-and-hold circuit SH4j and SH8j to sample and hold that yj value. The sequencer SEQ may be designed to accommodate the particular bit-serial format being used. The common bit-serial formats arrange the bits in order of their significance, one such format beginning each digital word with its least significant bit and ending each digital word with its most significant (sign) bit, and another such format beginning each digital word with its most significant (sign) bit and ending each digital word with its least significant bit.

At the end of the word interval, the four partial weighted summation results yj are held in parallel at the output ports of the sample-and-hold circuits SH1j, SH2j, SH3j and SH4j and the four complementary partial weighted summation results -yj are held in parallel at the output ports of the sample-and-hold circuits SH5j, SH6j, SH7j and SH8j. Weighting capacitors C1j, C2j, C3j and C4j are used to weight the four partial weighted summation results held by the sample-and-hold circuits SH1j, SH2j, SH3j and SH4j, respectively. The four partial weighted summation results held by the the sample-and-hold circuits SH1jj, SH2j, SH3j and SH4j are weighted in 1:2:4:8 ratio by weighting capacitors C1j, C2j, C3j and C4j, in accordance with the amplitudes of the weights +1, +2, +4 and -8 of the respective groups of bits generating them. (The most significant group of bits are assumed to be sign bits of two's complement numbers, which is the reason for the negative sign of the weighting by eight.) Similarly, the four partial weighted summation results held by the the sample-and-hold circuits SH5j, SH6j, SH7j and SH8j are weighted in 1:2:4:8 ratio by weighting capacitors C5j, C6j, C7j and C8j, in accordance with the amplitudes of the weights +1, +2, +4 and -8 of the respective groups of bits generating them. Accordingly, weighting capacitors C1j, C2j, C3j, C4j, C5j, C6j, C7j, and C8j have respective capacitances in 1:2:4:8:1:2:4:8 ratio. The four partial weighted summation results held by the the sample-and-hold circuits SH1j, SH2j, SH3j and SH4j charge the weighting capacitors C1j, C2j, C3j and C4j in respective amounts depending on their voltages and the capacitances of the capacitors to which they are applied, in accordance with Coulomb's Law. The four partial weighted summation results held by the the sample-and-hold circuits SH5j, SH6j, SH7j and SH8j charge the weighting capacitors C5j, C6j, C7j and C8j in respective amounts depending on their voltages and the capacitances of the capacitors to which they are applied, in accordance with Coulomb's Law.

A charge sensing amplifier QS(N+j) then differentially senses the charge on the plates of the weighting capacitors C5j, C6j, C7j and C4j connecting to its inverting (-) input connection and the combined charge on the plates of the weighting capacitors C1j, C2j, C3j and C8j connecting to its non-inverting (+) input connection, to generate a final weighted summation result as a sample of its output signal zj. Connecting plates of the weighting capacitors C4j and C8j to the inverting (-) input connection of the charge sensing amplifier QS(N+j) and to its non-inverting (+) input connection, respectively, provides for the negativeness associated with the minus eight weight of the sign bit in the two's complement numbering system.

Reset or zeroing of the charge sensing amplifiers QS(N+j) in FIGURE 5 takes place at the same time as the reset or zeroing of the charge sensing amplifiers QSj in the preceding FIGURE 2, 3 or 4 weighted summation circuitry. During reset the (VSS + VDD)/2 third operating voltage at the yj and -yj output connections of the charge sensing amplifiers QSj are applied to the plates of the weighting capacitors C1j, C2j, C3j, C4j, C5j, C6j, C7j, and C8j by the sample-and-hold circuits SH1j, SH2j, SH3j, SH4j, SH5j, SH6j, SH7j and SH8j.

Where the input signals to the FIGURE 2, FIGURE 3 or FIGURE 4 weighted summation apparatus comprise plural-bit words, the N signals z&sub1;, z&sub2;,...zN from a succeeding FIGURE 5 weighted summation apparatus can then, per FIGURE 6, be applied to an array of respective non-linear processors NLP&sub1;, NLP&sub2;,... NLPN to generate a plurality, N in number, of analog axonal output responses. As shown in FIGURE 6, these analog axonal output responses can be digitized in respective analog-to-digital converters ADC&sub1;, ADC&sub2;, ADC&sub3;,...ADCN, for application to the succeeding circuitry NEXT. For example, this succeeding circuitry may be an output neural net layer where the neural net layer thusfar described is a hidden neural net layer.

FIGURE 7 shows non-linear voltage amplifier circuitry that can be used to implement each non-linear processor NLPj in FIGURE 6. The FIGURE 7 non-linear voltage amplifier is a cascade connection of two source-follower transistors, one (Q20A) being an n-channel MOSFET and the other (Q20B) being a p-channel MOSFET. Q20A is provided a constant-current generator source load by an n-channel MOSFET Q&sub2;&sub1;, which is the slave or output transistor of a current mirror amplifier including as its master or input transistor an n-channel MOSFET Q&sub2;&sub2; self-biased by drain-to-gate feedback. Q20B is provided a constant-current generator source load by a p-channel MOSFET Q&sub2;&sub3;, which is the slave or output transistor of a current mirror amplifier including as its master or input transistor a p-channel MOSFET Q&sub2;&sub4; self-biased by drain-to-gate feedback. Q&sub2;&sub2; and Q&sub2;&sub4; are connected as diodes by their respective drain-to-gate feedback connections, and these diodes are connected in series with another diode-connected n-channel MOSFET Q&sub2;&sub5; and with another diode-connected p-channel MOSFET Q&sub2;&sub6; between VSS and VDD potentials to implement a bias network. In this bias network a quiescent input current flows from the input port of the current mirror amplifier comprising Q&sub2;&sub3;, Q&sub2;&sub4; into the input port of the current mirror amplifier comprising Q&sub2;&sub1;, Q&sub2;&sub2;. Q&sub2;&sub1; and Q&sub2;&sub3; drain current flows are similar-valued by current mirror amplifier action.

All the n-channel MOSFETs Q20A, Q&sub2;&sub1;, Q&sub2;&sub2; and Q&sub2;&sub5; have similar channel widths and lengths and exhibit similar operating characteristics. All the p-channel MOSFETs Q20B, Q&sub2;&sub3;, Q&sub2;&sub4; and Q&sub2;&sub6; have similar channel widths and lengths and exhibit similar operating characteristics, which are complementary to those of the n-channel MOSFETs. The bias network MOSFETs Q&sub2;&sub2;, Q&sub2;&sub4;, Q&sub2;&sub5; and Q&sub2;&sub6; may be shared by a plurality of the FIGURE 7 non-linear voltage amplifier circuits to conserve hardware and operating power.

Non-linearity of response in the FIGURE 7 voltage amplifier comes about because (1) source-follower action of Q20A for positive-going excursions of its gate electrode potential becomes limited as its source potential approaches its drain potential VHI and (2) source-follower action of Q20B for negative-going excursions of its gate electrode potential becomes limited as its source potential approaches its drain potential VLO. At the source electrode of source-follower Q20B there is a sigmoidal response to a linear ramp potential applied to the gate electrode of source-follower Q20A. The voltages VLO and VHI can be programmed to control the limiting properties of the FIGURE 7 non-linear amplifier, and the voltages VLO and VHI may be selected to provide for symmetry of response or for asymmetry of response. FIGURE 7 shows representative values for VHI and VLO that provide a substantially symmetrical response about +2.5 volts.

Performing the final weighted summation processes in the analog regime, as thusfar described, facilitates also performing in the analog regime the sigmoidal non-linear processing subsequently required in a neural network. Usually, a further array of capacitive structures is required for storing the partial weighted summation results, then performing the final weighted summation in the analog regime. Performing the final weighted summation processes in the digital regime avoids the need for such a further array of capacitive structures. A digital accumulator circuit is used instead, which tends to be more economical of area on a monolithic integrated-circuit die. In a neural network, performing the sigmoidal non-linear processing in the digital regime tends to be less straightforward. However, the determination of the slope of the non-linear transfer function, which determination is necessary when training a neural net layer using a back-propagation training algorithm, is simply accomplished in certain digital non-linear circuitry as will be described further on in this specification. This contrasts with the greater difficulty of determining the slope of the non-linear transfer function in analog non-linear circuitry.

FIGURES 9A and 9B when arranged as shown in FIGURE 9 provide a schematic diagram of a neural net layer wherein the the final weighted summation processes are performed in the digital regime. The weighting capacitors of the FIGURE 9 neural net layer are preferably constructed on a binary-digit-sliced basis as described in allowed U. S. patent application serial No. 525,931, filed 21 May 1990 by W. E. Engeler, entitled "WEIGHTED SUMMATION CIRCUITS HAVING DIFFERENT-WEIGHT RANKS OF CAPACITIVE STRUCTURES" and assigned to General Electric Company. In FIGURE 9A the bit-sliced (column) input signals x&sub1;, x&sub2;,... xM are applied to a weighted summation network WSN&sub1; that comprises a rank of capacitive weighting structures of relatively high significance; and in FIGURE 9B the bit-sliced (column) input signals x&sub1;, x&sub2;,... xM are applied to a weighted summation network WSN&sub3; that comprises a rank of capacitive weighting structures of relatively low significance. In FIGURE 9A the analog weighted sums of the bit-sliced input signals x&sub1;, x&sub2;,... xM supplied from the weighted summation network WSN&sub1;, which are of relatively higher significances, are digitized in respective analog-to-digital (or A-to-D) converters RADC&sub1;, RADC&sub2;, RADC&sub3;,...RADCN to generate bit-sliced digital signals Y&sub1;, Y&sub2;, Y&sub3;,...YN, respectively. Analogously, in FIGURE 9B the analog weighted sums of the bit-sliced input signals x&sub1;, x&sub2;,... xM supplied from the weighted summation network WSN&sub3;, which are of relatively lower significances, are digitized in respective analog-to-digital (or A-to-D) converters RADC(N+1), RADC(N+2), RADC(N+3), ...RADC2N to generate bit-sliced digital signals y(N+1), Y(N+2), Y(N+3), ...Y2N, respectively.

In FIGURE 9A the bit-sliced digital signals y&sub1;, y&sub2;, y&sub3;, ...yN, which are partial weighted summations, are applied to a weighted summation network WSN&sub2; (as will be described in greater detail further on with reference to FIGURE 8). The weighted summation network WSN&sub2; combines each set of partial weighted summations yj to generate a respective relatively more significant component Syj of the final weighted summation supplied as the sum signal output of a digital adder ADDj, which respective relatively more significant component Syj is applied as augend input signal to the adder ADDj. Analogously, in FIGURE 9B the bit-sliced digital signals y(N+1), YN+2), Y(N+3), ...Y2N, which are partial weighted summations, are applied to a weighted summation network WSN&sub4; (as will be described in greater detail further on with reference to FIGURE 8). The weighted summation network WSN&sub4; combines each set of partial weighted summations y(N+j) to generate a respective relatively less significant component Σy(N+j) of the final weighted summation supplied as the sum signal output of a digital adder ADDj, which respective relatively less significant component Σy(N+j) is applied after appropriate wired shift right WSRj as addend input signal to the adder ADDj. (Right shift of a digital signal by P binary places is reckoned as being a division of the digital signal by 2P.)

The sum signals from the adders ADD&sub1;, ADD&sub2;, ADD&sub3;, ... ADDN are supplied as input signals to respective digital non-linear processors NLP&sub1;, NLP&sub2;, NLP&sub3;,... NLPN which respond to their respective input signals to generate a plurality, N in number, of digital axonal output responses. As shown in FIGURE 9A, these digital axonal output responses can be applied to succeeding circuitry NEXT. For example, this succeeding circuitry NEXT may be an output neural net layer where the neural net layer thusfar described is a hidden neural net layer.

Where the input signals to the FIGURE 2, FIGURE 3 or FIGURE 4 weighted summation apparatus comprise single-bit words, the N signals y&sub1;, y&sub2;, y&sub3;,...yN can, per FIGURE 9A, be digitized in respective analog-to-digital (or A-to-D) converters RADC&sub1;, RADC&sub2;, RADC&sub3;,...RADCN, for application to adders ADD&sub1;, ADD&sub2;, ADD&sub3;, ... ADDN, which supply their sum signals to respective digital non-linear processors NLP&sub1;, NLP&sub2;, NLP&sub3;,... NLPN to generate a plurality, N in number, of digital axonal output responses. There is no need for the further, digital weighted summation apparatus WSN&sub2;. Analogously,the N signals y(N+1), y(N+2), y(N+3),...y2N can, per FIGURE 9B, be digitized in respective analog-to-digital (or A-to-D) converters RADC(N+1), RADC(N+2), RADC(N+3), ...RADC2N, for application with appropriate wired shifts right WSR&sub1;, WSR&sub2;, WSR&sub3;, ... WSRN to adders ADD&sub1;, ADD&sub2;, ADD&sub3;, ... ADDN. There is no need for the further, digital weighted summation apparatus WSN&sub4;.

Where only a single rank of capacitive weighting structures -- that rank in the weighted summation network WSN&sub1; -- is used, all the elements shown in FIGURE 9B are dispensed with. The adders ADD&sub1;, ADD&sub2;, ADD&sub3;, ... ADDN are supplied zero-valued addend signals rather than being supplied Σy(N+1), Σy(N+2), Σy(N+3), ... Σy2N as respective addend signals. Alternatively, the adders ADD&sub1;, ADD&sub2;, ADD&sub3;, ... ADDN may be omitted, with the digital non-linear processors NLP&sub1;, NLP&sub2;, NLP&sub3;,... NLPN receiving their respective input signals directly from the weighted summation network WSN&sub2;. Or directly from the A-to-D converters RADC(N+1), RADC(N+2), RADC(N+3), ...RADC2N where the input signals x&sub1;, x&sub2;,... xM to the weighted summation network WSN&sub1; comprise single-bit words so the weighted summation network WSN&sub1; is not used.

FIGURE 8 shows further, digital weighted summation apparatus that can be used with the analog weighted summation apparatus of FIGURE 2, 3 or 4 to provide for the weighted summation of digital input signals x&sub1;, x&sub2;,... xM having plural-bit words. The digital signals are placed in bit-serial form and in word alignment for application to the weighted summation apparatus of FIGURE 2, 3 or 4, which processes the bit-serial digital signals on a bit-slice basis to generate B partial weighted summation results, B being the number of bits per bit-serial arithmetic word. The successive bit slices of the plural-bit words of the digital input signals x&sub1;, x&sub2;,... xM are processed in order of increasing significance, with the sign bit slice being processed last in each word. A SIGN BIT FLAG that is normally ZERO is provided, which SIGN BIT FLAG goes to ONE to signal the parallel occurrence of the sign bits in each successive group of M input signal words at the time the partial weighted summation of greatest significance is being generated, supposing the digital signals to be flows of two's complement arithmetic words.

FIGURE 8 shows the plurality N in number of A-to-D converters RADC&sub1;, RADC&sub2; ... RADCN used to convert the analog y&sub1;, y&sub2;, ... yN partial weighted summations from the charge sensing amplifiers RQS&sub1;, RQS&sub2;, RQS&sub3;, ... RQSN to digital format and a weighted summation network for combining the sequences of digitized y&sub1;, y&sub2;, ... yN partial weighted summations to generate final weighted summations Σy&sub1;, Σy&sub2;, ..., ΣyN. More particularly, the digitized y&sub1;, y&sub2;, ... yN partial weighted summations from the FIGURE 8 A-to-D converters RADC&sub1;, RADC&sub2; ... RADCN are assumed to be respective two's complement numbers. The bit resolution afforded by the A-to-D converters RADC&sub1;, RADC&sub2; ... RADCN preferably increases by one bit place each successive bit slice of the digital word; in any case, each successive partial weighted summation within the same digital word is weighted by an additional factor of two respective to the previous partial weighted summation. A plurality, N in number, of accumulators RACC&sub1;, RACC&sub2;, ... RACCN are used for generating respective ones of N respective final weighted summation results Σyj. Each of the successive partial weighted summation results yj sequentially supplied by the A-to-D converter RADCi is routed through a respective selective complementor RCMPi. The selective complementor RCMPi complements each bit of the partial weighted summation result the A-to-D converter RADCi generates when the SIGN BIT FLAG is ONE, then augments the result of this one's complementing procedure by adding to it a number that has a ONE as its least significant bit and has more significant bits all of which are ZEROs. This augmentation is simply carried out by applying the SIGN BIT FLAG as a carry to the adder in the accumulator RACCi. When the SIGN BIT FLAG is ZERO, the selective complementor RCMPi transmits to the accumulator RACCi, without complementation, each bit of the partial weighted summation result the A-to-D converter RADCi generates. Each of these accumulators RACC&sub1;, RACC&sub2;, ... RACCN accumulates for each word the successive partial weighted summation results yj sequentially supplied as one of the output signals of the weighted summation apparatus of FIGURE 2, 3 or 4, to generate a final weighted summation result Σyj. (In alternative embodiments of the invention where the bit-serial digital input signals supplying the parallel bit streams x&sub1;, x&sub2;, x&sub3;, ... xM are invariably of one polarity and are processed as unsigned numbers, of course, the selective complementors RCMP&sub1;, RCMP&sub2;, ... RCMPN need not be included in the respective connections of the A-to-D converters RADC&sub1;, RADC&sub2; ... RADCN to the accumulators RACC&sub1;, RACC&sub2;, ... RACCN.)

The respective Σyj final weighted summation result generated by each accumulator RACCi is latched by a respective clocked latch RLCHi to be temporarily stored until such time as the next final weighted summation result Σyj is generated by the accumulator RACCi. The respective Σyj final weighted summation result from latch RLCHi is supplied as augend input signal to a respective digital adder ADDj, which may receive as an addend a final weighted summation result Σyj&min; of lesser significance from a further, less significant rank of capacitors for weighting the plurality M in number of parallel bit streams x&sub1;, x&sub2;,... xM, which final weighted summation result Σyj' of lesser significance is appropriately attenuated respective to Σyj by a wired shift right WSRi. The sum output signal zi from the adder ADDi is supplied to the ensuing digital non-linear processor NLPi; and , as shown in FIGURE 6A, the non-linear response from the digital nonlinear processor NLPi is applied to the succeeding circuitry NEXT.

Where the input signals to the FIGURE 2, FIGURE 3 or FIGURE 4 weighted summation apparatus comprise plural-bit words, the N signals z&sub1;, z&sub2;,...zN from a succeeding FIGURE 8 weighted summation apparatus can then, per FIGURE 9, be applied to an array of respective digital non-linear processors NLP&sub1;, NLP&sub2;,... NLPN to generate a plurality, N in number, of digital axonal output responses for application to the succeeding circuitry NEXT. For example, this succeeding circuitry may be an output neural net layer where the neural net layer thusfar described is a hidden neural net layer.

FIGURE 10 shows circuitry that can be used to implement a digital non-linear processor NLPj as may be used for each of the digital non-linear processors NLP&sub1;, NLP&sub2; , NLP&sub3;, ... NLPN of FIGURE 9. The FIGURE 10 digital non-linear processor is used to obtain a transfer function as diagrammed in FIGURE 11. The FIGURE 11 transfer function approximates a sigmoidal symmetrically non-linear response by five straight line segments 11, 13, 15, 17 and 19. One skilled in the art of digital circuit design will after being acquainted with the following description of the FIGURE 10 digital non-linear processor be able to design more complex digital non-linear processors approximating a sigmoidal transfer characteristic using an odd number greater than five of straight line segments.

In the FIGURE 11 graph of the transfer function of the FIGURE 10 digital non-linear processor, the straight line segment 15 passes through zero input signal, extending between inflection points 14 and 16 which occur at input signals of values -k&sub1; and k&sub1;, respectively. The input signal has relatively small absolute magnitudes in the range defined by the straight line segment 15, which segment of the transfer characteristic has the steepest slope and passes through the origin of the transfer characteristic graph.

The straight line segment 11 extending left and down from inflection point 12 and the straight line segment 19 extending right and up from inflection point 18 have the shallowest slope. The input signal has relatively large absolute magnitudes in the ranges defined by the straight line segments 11 and 19. If the straight line segment 11 were extended without Change in its direction right and up from inflection point 12 it would cross the input axis at an intercept value of -k&sub3;; and if the straight line segment 19 were extended without change in its direction left and down from inflection point 18 it would cross the input axis at an intercept value of +k&sub3;.

The input signal has intermediate absolute magnitudes in the ranges defined by the straight line segments 13 and 17. The straight line segment 13 extends between the inflection points 12 and 14 and, if it were extended without change in its direction beyond inflection point 14, would cross the output signal axis at an intercept value of -k&sub4;. The straight line segment 17 extends between the inflection points 16 and 18 and, if it were extended without change in its direction beyond inflection point 16, would cross the output signal axis at an intercept value of +k&sub4;.

In the FIGURE 7 digital non-linear processor the digital final weighted summation result Σyj is supplied to a respective absolute value circuit AVCj where the most significant, sign bit conditions a selective complementor RCMP(N+j) to complement the less significant bits of Σyj when zj is negative. The output signal from the selective complementor RCMP(N+j) is supplied as augend signal to a digital adder ADD(N+j), which is supplied the most significant bit of Σyj as an addend signal. The sum output of the adder ADD(N+j) is the |Σyj| output signal of the absolute value AVCj.

A digital subtractor SUBj is used for subtracting a positive constant value k&sub1; from |Σyj| to determine whether |Σyj| being < k&sub1; should be on a portion of the non-linear transfer characteristic having steepest slope or whether |Σyj| being ≧ k&sub1; should be on a portion of the non-linear transfer characteristic having a shallower slope. If the most significant, sign bit of the difference output signal of the subtractor SUBi is a ONE, indicating |Σyj| < k&sub1;, a slope multiplexer SMXj selects the steepest transfer characteristic slope as the multiplier signal for a digital multiplier MULTj that is supplied |Σyj| as multiplicand signal. The digital product is then added in a digital adder ADD(2N+j) to a zero-value intercept, as selected by an intercept multiplexer IMXj responsive to the sign bit of the difference output signal of subtractor SUBi being a ONE, to generate the non-linear response |zj| to |Σyj|.

A digital subtractor SUB(N+j) is used for subtracting a positive constant value k&sub2; from |Σyj| to determine whether |Σyj| being ≧ k&sub2; should be on a portion of the non-linear transfer characteristic having shallowest slope or whether |Σyj| being < k&sub2; should be on a portion of the non-linear characteristic having a steeper slope. It is most significant, sign bit of the difference output signal of the subtractor SUB(N+j) is a ZERO, indicating |Σyj| ≧ k&sub2;, the slope multiplexer SMXj selects the shallowest transfer characteristic slope as the multiplier signal for the digital multiplier MULTj; and the intercept multiplexer IMXj selects an intercept of value k&sub3; somewhat smaller than k&sub2; to be added to the product from multiplier MULTj in adder ADD(N+j) to generate |zj|.

Responsive to the most significant bits of the difference signals from the subtractors SUBj and SUB(N+j) being ZERO and ONE respectively, indicative that |Σyj| should be on a portion of the non-linear transfer characteristic have a slope intermediate between the steepest and shallowest slopes, the slope multiplexer SMXj selects that intermediate slope as multiplier signal to the multiplier MULTj, and the intercept multiplexer IMXj selects an intercept of value k&sub4; somewhat smaller than k&sub1; as the addend signal supplied to the adder ADD(2N+j).

The non-linear response |zj| to |Σyj| is converted to zj response to Σyj by processing |zj| through a selective complementor CMP(2N+j) that complements each bit of |zj| when the most significant, sign bit of Σyj is a ONE indicating its negativity; inserting the most significant bit of Σyj as sign bit before the selective complementor CMP(2N+j) output signal; and adding the resultant in a digital adder ADD(3N+j) to the most significant bit of Σyj. The sum signal from the digital ADD(3N+j) is then used to load a parallel-in/serial-out register RPISOj for the jth row, which supplies zj in bit-serial format to the succeeding circuitry NEXT (per FIGURE 6).

During back-propagation training, the value of the slope selected by the slope multiplexer SMXj responsive to the |Σyj| generated by a set of prescribed synapse input signals can be temporaily stored in a latch SLCHj responsive to a LATCH SLOPE command. This operation will be considered as part of an overall description of back-propagation training further on in this specification. In FIGURES 20-23 and 27-31, the slope latch SLCHj is shown external to the digital non-linear processor NLPj with which it is associated.

Thusfar, the reversal of sign in the weighting of the parallel sign bits of the bit-serial synapse input signals has been described as being done in the final weighted summation of the partial weighted summation results. Alternatively, in other embodiments of the invention, such as those now to be described, the reversal of sign in the weighting of the parallel sign bits of the bit-serial synapse input signals is done while performing the partial weighted summations.

FIGURE 12 shows a modification of the FIGURE 2 or FIGURE 3 apparatus in which the multiplexers MX&sub1; ... MXM, MX(M+1) ... MX2M that each select from between two operating voltages the voltage applied to a corresponding one of the input lines IL&sub1; ... ILM, IL(M+1) ... IL2M are replaced by multiplexers MX&sub1;&min; ... MXM&min;, MX(M+1)&min; ... MX2M&min; that each select from among the VDD, VSS and (VSS + VDD)/2 operating voltages the voltage applied to a corresponding one of the input lines IL&sub1; ... ILM, IL(M+1) ... IL2M. The current condition of the SIGN BIT FLAG is applied to each of the multiplexers MXi&min; and MX(i+M)&min; as its first control bit, and the current bit of a respective input voltage signal xi is applied to each of the multiplexers MXi&min; and MX(i+M)&min; as its second control signal.

For all bits of xi except its sign bits, the SIGN BIT FLAG is a ZERO. The SIGN BIT FLAG being a ZERO conditions multiplexer MXi&min; to respond to xi being a ONE to apply the VDD first operating voltage to an input line ILi and to respond to xi being a ZERO to apply the third operating voltage (VSS + VDD)/2 to the input line ILi. The SIGN BIT FLAG being a ZERO conditions multiplexer MX(i+M)&min; to respond to xi being a ONE to apply the VSS second operating voltage to an input line IL(i+M) and to respond to xi being a ZERO to apply the third operating voltage (VSS + VDD)/2 to the input line IL(i+M).

When the sign bits of xi occur, the SIGN BIT FLAG is a ONE. The SIGN BIT FLAG being a ONE conditions multiplexer MXi&min; to respond to xi being a ONE to apply the VSS second operating voltage to an input line ILi and to respond to xi being a ZERO to apply the third operating voltage (VSS + VDD)/2 to the input line ILi. The SIGN BIT FLAG being a ONE conditions multiplexer MX(i+M)&min; to respond to xi being a ONE to apply the VDD first operating voltage to an input line IL(i+M) and to respond to xi being a ZERO to apply the third operating voltage (VSS + VDD)/2 to the input line IL(i+M). Accordingly,the reversal of sign in the weighting of the parallel sign bits of the bit-serial synapse input signals is done while performing the partial weighted summations.

FIGURE 13 shows a modification of the FIGURE 4 apparatus in which the multiplexers MX&sub1;, MX&sub2;, MX&sub3;, ... MXM, that each select from between two operating voltages the voltage applied to a corresponding one of the input lines IL&sub1;, IL&sub2;, IL&sub3;, ... ILM are replaced by multiplexers MX&sub1;&min;, MX&sub2;&min;, ... MXM&min; that each select from among the VDD, VSS and (VSS + VDD)/2 operating voltages the voltage applied to a corresponding one of the input lines IL&sub1;, IL&sub2;, IL&sub3;, ... ILM. The multiplexers MX&sub1;&min;, MX&sub2;&min;, ... MXM&min; are controlled in the FIGURE 13 apparatus similarly to the way they are in FIGURE 12 apparatus, again to provide the reversal of sign in the weighting of the parallel sign bits of the bit-serial synapse input signals while performing the partial weighted summations.

FIGURE 14 shows analog apparatus for performing the final weighted summation of partial weighted summation results from the FIGURE 12 or 13 apparatus. The FIGURE 14weighted summation apparatus can be described as weighted accumulation apparatus since it accumulates the B partial weighted summation results supplied to it from the weighted summation apparatus of FIGURE 12 or 13, the accumulated partial weighted summation result term being weighted in accordance with the significance of the bit slice involved in its generation. The FIGURE 14 apparatus differs from the FIGURE 5 apparatus in that, for each value of j, one through N, the weighting capacitors C1j, C2j, C3j and C4j all connect to a single respective output line OLj , so the sum of their respective charge conditions can be sensed on a single-ended basis. Single-ended charge sensing allows dispensing with the weighting capacitors C5j, C6j, C7j and C8j. Single-ended charge sensing will also allow each of the charge sensing amplifiers QS(N+j) to be replaced by a simpler single-ended charge sensing amplifier such as that shown in FIGURE 1. Where the FIGURE 14 apparatus performs the final weighted summation of partial weighted summation results from the FIGURE 12 apparatus, each of the charge sensing amplifiers QSj can also be replaced by a simpler single-ended charge sensing amplifier such as that shown in FIGURE 1.

FIGURE 15 is a schematic diagram of a neural net layer similar to the neural net layer of FIGURE 6, but includes either of the FIGURE 12 and FIGURE 13 apparatuses as a weighted summation network WSN&sub1;&min; for performing a plurality of weighted summation procedures, so the reversal of sign in the weighting of the parallel sign bits of the bit-serial synapse input signals takes place in the weighted summation network WSN&sub1;&min;. Where the weighted summation procedures performed in the weighted summation network WSN&sub1;&min; generate partial weighted summation results, a weighted summation network WSN&sub2;&min; comprising the FIGURE 14 apparatus is used to generate final weighted summation results for application to an array of non-linear processors NLP&sub1;, NLP&sub2;, ... NLPN. The non-linear processors NLP&sub1;, NLP&sub2;, ... NLPN have their analog axonal output signals digitized in respective analog-to-digital converters ADC&sub1;, ADC&sub2;,...ADCN, for application to the succeeding circuitry NEXT.

FIGURE 16 shows a modification made to the FIGURE 12 apparatus in the FIGURE 15 neural net layer in respect to each synapse input signal port. These modifications are made to adapt the FIGURE 15 neural net layer for operation with one's complement, rather than with two's complement, bit-serial synapse input signals. The one's complement bit-serial synapse input signals are presumed to be ones in which the sign bit is the initial bit in a bit-serial word. Responsive to the SIGN BIT FLAG being a ONE, a respective bit latch SBLi latches the sign bit of xi to be held for the remainder of the bit-serial word when the SIGN BIT FLAG is a ZERO. Another bit latch ABLi clocked at bit rate latches each successive bit of xi. The outputs of the bit latches SBLi and ABLi are applied as the two inputs of a two-input exclusive-OR gate XORi, if the sign bit stored in sign bit latch SBLi is a ZERO indicating xi to be positive, the exclusive-OR gate XORi applies bits corresponding to the successive bits of xi as second control bits to the multiplexers MXi&min; and MX(i+M)&min;. Alternatively, if the sign bit stored in sign bit latch SBLi is a ONE indicating xi to be negative, the exclusive-OR gate XORi applies bits corresponding to the bit complements of the successive bits of xi as second control bits to the multiplexers MXi&min; and MX(i+M)&min;.

When the SIGN BIT FLAG is a ONE during the initial bit of each bit-serial word, the bit latches SBLi and ABLi latch the same sign bit to each of the two inputs of exclusive-OR gate XORi, which responds with a ZERO output. This ZERO output is applied as the second control bit of each of the multiplexers MXi&min; and MX(i+M)&min;, conditioning them to select (VSS + VDD)/2 to each of the input lines ILi and IL(i+M). This facilitates the resetting of the charge sensing amplifiers QSN+1), QSN+2), ... QS2N during the sign bit interval.

When adapting the FIGURE 15 neural net layer for operation with one's complement bit-serial synapse input signals, the FIGURE 14 sequencer SEQ is modified to omit weighting capacitors SH4j for the sign bit slice and the associated sample-and-hold circuits SH4j. The FIGURE 14 sequencer SEQ in the FIGURE 15 neural net layer is also arranged to sequence sampling in the sample-and-hold circuits in accordance with the significances of the successive bits of the bit-serial xi words which follow.

FIGURE 17 shows a modification made to the FIGURE 12 apparatus in the FIGURE 15 neural net layer in respect to each synapse input signal port to facilitate operation with one's complement bit-serial synapse input signals in which the final bit in each bit-serial word is the sign bit. It differs from the FIGURE 16 modification in that xi is delayed for the duration of one bit-serial word in a digital delay line DDLi before its application to the exclusive-OR gate XORi bit latch ABLi, so the sign bit latch SBLi can be written for temporarily storing the final sign bit of each bit-serial word to be applied to one input of the exclusive-OR gate XORi throughout the interval that the other bits of that bit-serial word are successively applied to the other input of the exclusive-OR gate XORi. The clocking of the digital delay line DDLi and of the sign bit latch SBLi in FIGURE 17 is arranged so that the exclusive-OR gate XORi invariably provides a ZERO for application as second control bits to the multiplexers MXi&min; and MX(i+M)&min; during the time the SIGN BIT FLAG is a ONE, facilitating the resetting of the charge sensing amplifiers QS(N+1), QS(N+2), ... QS2N. Again, the FIGURE 14 sequencer SEQ in the FIGURE 15 neural netlayer is modified to omit weighting capacitors SH4j for the sign bit slice and the associated sample-and-hold circuits SH4j. The FIGURE 14 sequencer SEQ in the FIGURE 15 neural net layer is also arranged to sequence sampling in the sample-and-hold circuits in accordance with the significances of the successive bits of the bit-serial xi words which follow.

FIGURE 18 shows apparatus for performing the final weighted summation of partial weighted summation results from the FIGURE 12 or 13 apparatus digitally after digitizinging the analog partial weighted summation results. The FIGURE 18 apparatus differs from the FIGURE 8 apparatus in that: there are no selective complementors RCMP&sub1;, RCMP&sub2;, ... RCMPN included in the respective connections of the A-to-D converters RADC&sub1;, RADC&sub2; ... RADCN to the accumulators RACC&sub1;, RACC&sub2;, ... RACCN; and the accumulators invariably have ZERO carry inputs.

FIGURE 19 is a schematic diagram of a neural net layer similar to the neural net layer of FIGURE 9, but includes either of the FIGURE 12 and FIGURE 13 apparatuses as a weighted summation network WSN&sub1;&min; for performing a plurality of weighted summation procedures, so the reversal of sign in the weighting of the parallel sign bits of the bit-serial synapse input signals takes place in the weighted summation network WSN&sub1;&min;. Where the weighted summation procedures performed in the weighted summation network WSN&sub1;&min; generate partial weighted summation results, a weighted summation network WSN&sub2;&min; comprising the FIGURE 18 apparatus is used to generate final weighted summation results for application to an array of non-linear processors NLP&sub1;, NLP&sub2;, ... NLPN. The non-linear processors NLP&sub1;, NLP&sub2;, ... NLPN have their analog axonal output signals digitized in respective A-to-D converters RADC&sub1;, RADC&sub2;, ... RADCN, for application to the succeeding circuitry NEXT.

The nature of the analog-to-digital converters RADCi has thusfar not been considered in detail. Indeed, a wide variety of A-to-D converters can be used in implementing the invention. Some A-to-D converter structures are capable of accepting input signals in charge form, and in such a structure the charge sensing means and the analog-to-digital converting means specified in certain of the claims appended to this application will both be provided. The bit-slicing of input signals already slows data word throughput rate, so oversampling A-to-D converters, as of the sigma-delta type will not be favored over successive-approximation A-to-D converters unless quite slow data processing rates are acceptable. Where fairly fast data processing rates are required, flash A-to-D converters will be preferred oversuccessive-approximation A-to-D converters. The resistive ladder used to scale reference voltage in a flash A-to-D converter can be shared with other flash A-to-D converters within the same monolithic integrated circuit, economizing the amount of hardware associated with this approach to analog-to-digital conversion.

FIGURE 20, comprising component FIGURES 20A and 20B, shows a representative modification that can be manifoldly made to a neural net as shown in FIGURES 2, 8 and 9 or in FIGURES 3, 8 and 9. Each neural net layer comprises a capacitive network viewed as having a plurality M in number of columns of capacitive weights and a plurality N in number of rows of capacitive weights. The description that follows focuses on the intersection of an ith of these columns with a jth of these rows, where i is a column index from 1 to M and j is a row index from 1 to N.

A respective modification is made near each set of intersections of an output line OLj with input lines ILi and IL(M+i) driven by opposite senses of a synapse input signal xi. Such modifications together make the neural net capable of being trained. Each capacitor pair Ci,j and C(M+i),j of the FIGURE 2 or 3 portion of the neural net is to be provided by a pair of digital capacitors DCi,j and DC(M+i),j. Capacitors DCi,j and DC(M+i),j are preferably of the type described by W. E. Engeler in his allowed U. S. patent application serial No. 526,470 filed 21 May 1990, entitled "CAPACITIVE STRUCTURES FOR WEIGHTED SUMMATION, AS USED IN NEURAL NETS". The capacitances of DCi,j and DC(M+i),j are controlled in complementary ways by a digital word and its one's complement, as drawn from a respective word-storage element WSEi,j in an array of such elements located interstitially among the rows of digital capacitors and connected to form a memory. This memory may, for example, be a random access memory (RAM) with each word-storage element WSEi,j being selectively addressable by row and column address lines controlled by address decoders. Or, by way of further example, this memory can be a plurality of static shift registers, one for each column j. Each static shift register will then have a respective stage WSEi,j for storing the word that controls the capacitances of each pair of digital capacitors DCi,j and DC(M+i),j.

The word stored in word storage element WSEi,j may also control the capacitances of a further pair of digital capacitors DCi,(N+j) and DC(M+i),(N+j), respectively. Capacitors DCi,(N+j) and DC(M+i),(N+j) are also preferably of the type described by in U. S. patent application serial No. 526,470. The capacitors DCi,(N+j) and DC(M+i),(N+j) connect between "ac ground" and input lines ILi and IL(M+i), respectively, and form parts of the loading capacitors CLi and CL(M+i), respectively. The capacitances of DC(M+i),(N+j) and DCi,j are similar to each other and changes in their respective values track each other. The capacitances of DCi,(N+j) and DC(M+i),j are similar to each other and changes in their respective values track each other. The four digital capacitors DCi,j, DC(M+i), DCi,(N+j) and DC(M+i),(N+j) are connected in a bridge configuration having input terminals to which the input lines ILi and IL(M+i) respectively connect and having output terminals connecting to output line OLj and to ac ground respectively. This bridge configuration facilitates making computations associated with back-propagation programming by helping make the capacitance network bilateral insofar as voltage gain is concerned. Alternatively, where the computations for back-propagation programming are done by computers that do not involve the neural net in the computation procedures, the neural net need not include the digital capacitors DCi,(N+j) and DC(M+i),(N+j). These digital capacitors DCi,(N+j) and DC(M+i),(N+j) are not needed either if very large loading capacitors are placed on the output lines OLj, but this alternative undesirably reduces sensitivity of the row charge-sensing amplifier RQSj.

A respective column driver CDi applies complementary bit-sliced digital signals to each pair of input lines ILi and IL(M+i). The column driver CDi comprises the multiplexers MXi and MX(M+i) connected per FIGURE 2 or per FIGURE 3. When the neural net modified per FIGURE 20 is being operated normally, following programming, the &phis;p signal applied to a mode control line MCL is a logic ZERO. This ZERO conditions a respective input line multiplexer ILMi to connect the non-inverting output port of each column driver CDi to input line ILi. The &phis;p signal on mode control line MCL being a ZERO also conditions a respective input line multiplexer ILM(M+i) to connect the inverting output port of each column driver CDi to input line IL(M+i).

A ZERO on mode control line MCL also conditions each output line multiplexer OLMj of an N-numbered plurality thereof to select the output line OLj to the input port of a respective row charge-sensing amplifier RQSj that performs a charge-sensing operation for output line OLj. When &phis;p signal on mode control line MCLis a ZERO, the input signal xi induces a total change in charge on the capacitors DCi,j and DC(M+i),j proportional to the difference in their respective capacitances. The resulting displacement current flow from the input port of the row charge-sensing amplifier RQSj requires that there be a corresponding displacement current flow from the Miller integrating capacitor CIj in the row charge-sensing amplifier RQSj charging that capacitor to place thereon a voltage vj defined as follows.

The voltage Vj is supplied to an analog-to-digital converter RADCj for the jth row of the capacitive network, which digitizes voltage Vj to generate a digital signal yj for the jth row that is assumed to be a two's complement digital signal. The digitized Vj signal is supplied to a selective complementor RCMPj for the jth row, which passes the digitized Vj signal without change for each of the bit slices of the input signals x&sub1;, x&sub2;, ... xM except the sign bit slices. For the sign bit slices the selective complementor RCMPj one's complements the bits of the digitized Vj signal and adds unity thereto. The selective complementor RCMPj supplies its output signal to an accumulator RACCj for the jth row of the capacitive network. The accumulator RACCj accumulates the bit slices of each set of samples of the input signals x&sub1;, x&sub2;, ... xM as weighted for the output line OLj, sensed and digitized; the accumulator result Syj is latched by a respective row latch RLCHj for the jth row; and the accumulator RACCj is then reset to zero. At the same time the accumulator RACCj is reset to zero, to implement dc-restoration a reset pulse &phis;R is supplied to the charge sensing amplifier RQSj for the jth row and to the charge sensing amplifiers for the other rows. During the dc-restoration all xi are "zero-valued".

During training, the &phis;p signal applied to mode control line MCL is a logic ONE, which causes the output line multiplexer OLMj to disconnect the output line OLj from the charge sensing amplifier RQSj and to connect the output line OLj to receive a bit-sliced δj error term. This δj error term is generated by a row driver RDj responsive to the bit-serial digital product output signal of a digital multiplier BSMj, responsive to a signal Δj and to the slope stored in the slope latch SLCHj. The term Δj for the output neural net layer is the difference between zj actual value and its desired value dj. The term Δj for a hidden neural net layer is the Δj output of the succeeding neural net layer during the back-propagation procedure.

The row driver RDj is shown as a type having differential outputs which can be provided by a pair of multiplexers similar to input multiplexers MXi and MX(M+i) connected as in FIGURE 2 or as in FIGURE 3. Since output of only one sense is required to drive the single-ended output line OLj, as used in the neural net layers shown in FIGURES 20 and 27, however, it is preferable from the standpoint of simplifying hardware to use row drivers RDj that consist of but a single multiplexer. Indeed, the row driver function may be subsumed in the bit-serial multiplier BSMj. In the neural net layers shown in FIGURES 21, 23, 28 and 30 the row driver RDj must be of a type having differential outputs, as can be provided by a pair of multiplexers similar to input multiplexers MXi and MX(M+i) connected as in FIGURE 2 or as in FIGURE 3.

During training, the &phis;p signal applied to the mode control line MCL being a ONE also causes the input line multiplexers ILMi and ILM(M+i) to disconnect the input lines ILi and IL(M+i) from the column driver CDi output ports and connect them instead to the non-inverting and inverting input terminals of a column charge-sensing amplifier CQSi. Each successive partial summation of the weighted δ&sub1;, δ&sub2;, ...δN, signals induces a differential change in charge between input lines ILj and IL(M+i) proportional to (Ci,j - C(M+i),j, which differential change in charge is sensed using the column charge sensing amplifier CQSi, then digitized in a respective column analog-to-digital converter CADCi for the ith column. The digital output signal from the column A-to-D converter CQSi is supplied to a selective complementor CCMPi for the ith column, which responds to a SIGN BIT FLAG to selectively one's complement the bits of that digital signal as applied to an accumulator CACCi for the ith column. The selective complementor CCMPi otherwise supplies the accumulator CACCi an input signal the bits of which correspond to the bits of the digital output signal from the column A-to-D converter CQSi. The accumulator CACCi accumulates the successive partial summations of the weighted δ&sub1;, δ&sub2;, ...δN signals to generate the final weighted summation result Δi supplied to a parallel-in/serial-out register CPISOi for the ith column. The register CPISOi supplies the final weighted summation result Δi in bit-serial format for the bit-serial digital multiplier of the preceding neural net layer if such there be.

Resetting of the column charge sensing amplifier CQSi is normally done shortly after a ZERO to ONE transition appears in the &phis;p signal applied to mode control line MCL and may also be done at other times. This procedure corrects for capacitive unbalances on the input lines ILi and IL(M+i) during back-propagation computations that follow the resetting procedure.

FIGURE 21, comprising component FIGURES 21A and 21B, shows a representative modification that can be manifoldly made to a neural net as shown in FIGURES 4, 8 and 9. The four digital capacitors DCi,j, DC(M+i),j, DCi,(N+j) and DC(M+i),(N+j) are connected in a bridge configuration having input terminals connecting from the input line ILi and from a-c ground respectively and having output terminals connecting to output lines OLj and OL(N+j) respectively.

When the neural net layer per FIGURES 4, 8 and 9 is being operated normally, following programming, the &phis;p signal applied to a mode control line MCL is a logic ZERO. This ZERO on mode control line MCL conditions each output line multiplexer OLMj of an N-numbered plurality thereof to select the output line OLj to the inverting input terminal of a fully differential amplifier in an associated differential charge sensing amplifier RQSj for the jth row. This ZERO on mode control line MCL also conditions each output line multiplexer OLM(N+j) to the non-inverting input terminal of the fully differential amplifier in the associated differential charge sensing amplifier RQSj. When &phis;p signal on mode control line MCL is a ZERO, the input signal xi induces a total differential change in charge on the capacitors DCi,j and DCi,(N+j) proportional to the difference in their respective capacitances. The resulting displacement current flows needed to keep the input terminals of differential amplifier DAj substantially equal in potential requires that there be corresponding displacement current flow from the integrating capacitor CIj and CI(N+j) differentially charging those charging capacitors to place thereacross a differential voltage vj defined as follows.

The A-to-D converter RADCj for the jth row is of a type capable of digitizing the balanced output signal from the differential charge sensing amplifier RQSj. The selective complementor RCMPj, the accumulator RACCj, the row latch RLCHj for each jth row, the digital adder ADDj, the digital non-linear processor NLPj, the slope latch SLCHj, and the digital multiplier BSMj correspond with those elements as described in connection with FIGURE 20. The bit-serial product signal from the digital multiplier BSMj is supplied to the row driver RDj, which can take either of the forms the column driver CDi can take in FIGURE 20. During training, the &phis;p signal applied to the mode control line MCL being a ONE causes the output line multiplexers OLMj and OLM(N+j) to connect the output lines OLj and OL(N+j) for receiving from the row driver RDj complementary digital responses to the bit-serial product from the digital multiplier BSMj. The ONE on the mode control line MCL also causes each input line multiplexer ILMi to disconnect each input line ILi from the output port of the logic inverter INVi and connect it instead to supply charge on a single-ended basis to the input port of a column charge-sensing amplifier CQSi.

FIGURE 22, comprising FIGURES 22A and 22B, shows a representative modification that can be made manifold times to the neural net shown in FIGURES 12, 18 and 19 or in FIGURES 13, 18 and 19, for the programmable weighting of the capacitances used in performing weighted summations of synapse signals. In this type of neural net the signs of the digital signals being weighted and summed are taken into account in the partial weighted summation procedure performed on an analog basis, rather than being taken into account in the final weighted summation procedure performed on an digital basis as is the case in the type of neural net shown in FIGURE 20. Accordingly, respective straight-through digital bussing replaces the selective complementor RCMPj for each jth row and the selective complementor CCMPi for each ith column. Accordingly, also, each column driver CDi comprises the multiplexers MXi and MX(M+i) connected per FIGURE 12 or per FIGURE 13, rather than the multiplexers MXi and MX(M+i) connected per FIGURE 2 or per FIGURE 3.

The row driver RDj&min; is shown as a type having differential outputs which can be provided by a pair of multiplexers similar to input multiplexers MXi&min; and MX(M+i)&min; connected as in FIGURE 12 or as in FIGURE 13. Since output of only one sense is required to drive the single-ended output line OLj, as used in the neural net layers shown in FIGURES 22 and 29, however, it is preferable from the standpoint of simplifying hardware to use row drivers RDj&min; that consist of but a single multiplexer. In the neural net layers shown in FIGURES 24 and 31 the row driver RDj&min; must be of a type having differential outputs, as can be provided by a pair of multiplexers similar to input multiplexers MXi&min; and MX(M+i)&min; connected as in FIGURE 12 or as in FIGURE 13. The row driver RDj&min; provides in the neural net layers shown in FIGURES 22, 24, 29 and 31 for taking care of the sign bit in the digital product δi in the partial weighted summation procedure, so it need not be taken care of in the final weighted summation procedure. Accordingly, the selective complementors CCMPi for each ith column are replaced by respective straight-through bussing.

In a variant of the FIGURE 21 neural net layer not shown in the drawing, each row driver RDj is also replaced by a row driver RDj&min; that takes either of the forms the column driver CDi can take in FIGURE 22. In this variant, too, the row driver RDj&min; provides for taking care of the sign bit in the digital product δi in the partial weighted summation procedure, so it need not be taken care of in the final weighted summation procedure. Accordingly, the selective complementors CCMPi for each ith column are replaced by respective straight-through bussing.

FIGURE 23, comprising FIGURES 23A and 23B, shows a neural net layer in which both forward propagation and back propagation through a capacitive network are carried out with balanced signals, in which digital capacitors have their capacitance values programmed from respective word storage locations within a digital memory, and in which the signs of the digital signals being weighted and summed are taken into account in the final weighted summation procedure performed on a digital basis. The FIGURE 23 neural net layer shares features with both the FIGURE 20 and the FIGURE 21 neural net layers. A respective set of four digital capacitors DCi,j, DC(M+i),j, DCi,(N+j) and DC(M+i),(N+j) is used at each intersection of an ith column with a jth row of the array of sets of digital capacitors, for weighting of forward propagated input signals during periods of normal operation, and for weighting of back propagated error signals during training periods. Each set of four digital capacitors DCi,j, DC(M+i),j, DCi,(N+j) and DC(M+i),(N+j) have their capacitance values programmed by a word stored in a word storage element WSEi,j of an interstital memory array IMA. Each set of four digital capacitors DCi,j, DC(M+i),j, DCi,(N+j) and DC(M+i),(N+j) are connected in a bridge configuration having input terminals respectively connecting from paired input lines ILi and IL(M+i) as in FIGURE 20A and having output terminals respectively connecting to output lines OLj and OL(N+j) as in FIGURE 21A. During periods of normal operation, a respective column driver CDi that comprises the multiplexers MXi and MX(M+i) connected per FIGURE 2 or per FIGURE 3 applies complementary bit-sliced digital input signals to each pair of input lines ILi and IL(M+i), as in FIGURE 20A; and a charge sensing amplifier RQSj for each jth row of the array of sets of digital capacitors, is connected to differentially sense the charges on paired output lines OLj and OL(N+j) as in FIGURE 21B. During training periods a respective row driver RDj constructed similarly to each column driver CDi applies to each pair of output lines OLj and OL(N+j) complementary bit-serial responses to digital product signal from a bit-serial digital multiplier BSMj, as in FIGURE 21B; and a charge sensing amplifier CQSj for each ith column of the array of sets of digital capacitors is connected to differentially sense the charges on paired input lines ILi and IL(M+i), as in FIGURE 20A.

FIGURE 24, comprising FIGURES 24A and 24B, shows another neural net layer in which both forward propagation and back propagation through a capacitive network are carried out withbalanced signals, and in which digital capacitors have their capacitance values programmed from respective word storage locations within a digital memory. In the FIGURE 24 neural net layer the signs of the digital signals being weighted and summed are taken into account in the partial weighted summation procedure performed on an analog basis, rather than being taken into account in the final weighted summation procedure performed on a digital basis. The FIGURE 24 neural net layer shares features with both the FIGURE 21 and the FIGURE 22 neural net layers. A respective set of four digital capacitors DCi,j, DC(M+i),j, DCi,(N+j) and DC(M+i),(N+j) is used at each intersection of an ith column with a jth row of the array of sets of digital capacitors, for weighting of forward propagated input signals during periods of normal operation, and for weighting of back propagated error signals during training periods. Each set of four digital capacitors DCi,j, DC(M+i),j, DCi,(N+j) and DC(M+i),(N+j) have their capacitance values programmed by a word stored in a word storage element WSEi,j of an interstital memory array IMA. Each set of four digital capacitors DCi,j, DC(M+i),j, DCi,(N+j) and DC(M+i),(N+j) are connected in a bridge configuration having input terminals respectively connecting from paired input lines ILi and IL(M+i) as in FIGURE 22A and having output terminals respectively connecting to output lines OLj and OL(N+j) as in FIGURE 21A. During periods of normal operation, a respective column driver CDi that comprises the multiplexers MXi and MX(M+i) connected per FIGURE 12 or per FIGURE 13 applies complementary bit-sliced digital input signals to each pair of input lines ILi and IL(M+i), as in FIGURE 22A; and a charge sensing amplifier RQSj for each jth row of the array of sets of digital capacitors, is connected to differentially sense the charges on paired output lines OLj and OL(N+j) as in FIGURE 21B. During training periods a respective row driver RDj constructed similarly to each column driver CDi applies to each pair of output lines OLj and OL(N+j) complementary bit-serial responses to digital product signal from a bit-serial digital multiplier BSMj, as in FIGURE 21B; and a charge sensing amplifier CQSj for each ith column of the array of sets of digital capacitors is connected to differentially sense the charges on paired input lines ILi and IL(M+i), as in FIGURE 22A.

FIGURE 25 shows apparatuses for completing the back-propagation computations, as may be used with a neural net having layers as shown in FIGURES 2, 8 and 9 or in FIGURES 3, 8 and 9 modified manifoldly per FIGURE 20; or having layers as shown in FIGURES 4, 8 and 9 modified manifoldly per FIGURE 21; or having layers as shown in FIGURES 12, 18 and 19 or in FIGURES 13, 18 and 19 modified manifoldly per FIGURE 22; or having layers as shown in FIGURE 23; or having layers as shown in FIGURE 24. The weights at each word storage element WSEi,j in the. interstitial memory array IMA are to be adjusted as the i column addresses and j row addresses are scanned row by row, one column at a time. An address scanning generator ASG generates this scan of i and j addresses shown applied to interstitial memory array IMA, assuming it to be a random access memory. The row address j is applied to a row multiplexer RM that selects bit-serial δj to one input of a bit-serial multiplier BSM&sub0;, and the column address i is applied to a column multiplexer CM that selects bit-serial xi to another input of the multiplier BSM&sub0;.

Multiplier BSM&sub0; generates in bit-serial form the product xi δj as reduced by a scaling factor η, which is the increment or decrement to the weight stored in the currently addressed word storage element WSEij in the memory array IMA. A serial-in/parallel-out register SIPO is used to convert the ηxiδj signal from bit-serial to the parallel-bit form of the signals stored in the memory array IMA. The former value of weight stored in word storage element WSEij is read from memory array IMA to a temporary storage element, or latch, TS. This former weight value is supplied as minuend to a digital subtractor SUB, which receives as subtrahend η xi δj from the serial-in/parallel-out register SIPO. The resulting difference is the updated weight value which is written into word storage element WSEi,j in memory array IMA to replace the former weight value.

FIGURE 26 shows how trained neural net layers L&sub0;, L&sub1; and L&sub2; are connected together in a system that can be trained. Each of the neural net layers is similar to neural net layers as shown in FIGURES 2, 8 and 9 or in FIGURES 3, 8 and 9 modified manifoldly per FIGURE 20; or in FIGURES 4, 8 and 9 modified manifoldly per FIGURE 21; or in FIGURES 12, 18 and 19 or in FIGURES 13, 18 and 19 modified manifoldly per FIGURE 22; or in FIGURE 23; or in FIGURE 24. Each of the neural net layers has a respective back-propagation processor per FIGURE 25 associated therewith.

L&sub0; is the output neural net layer that generates zj output signals and is provided with a back-propagation processor BPP&sub0; with elements similar to those shown in FIGURE 25 for updating the weights stored in the interstitial memory array of L&sub0;. L&sub1; is the first hidden neural net layer which generates zh output signals supplied to the output neural net layer as its xi input signals. These zi output signals are generated by layer L&sub1; as its non-linear response to the weighted sum of its xh input signals. This first hidden neural net layer L&sub1; is provided with a back-propagation processor BPP&sub1; similar to BPP&sub0;. L&sub2; is the second hidden neural net layer, which generates zh output signals supplied to the first hidden neural net layer as its xh input signals. These zh output signals are generated by layer L&sub2; as its nonlinear response to a weighted summation of its xg input signals. This second hidden layer is provided with a back-propagation processor similar to BPP&sub0; and to BPP&sub1;.

FIGURE 26 presumes that the respective interstitial memory array IMA of each neural net layer L&sub0;, L&sub1;, L&sub2; has a combined read/write bus instead of separate read input and write output busses as shown in FIGURE 25. FIGURE 26 shows the Δj, Δi and Δh signals being fed back over paths separate from the feed forward paths for zj, zi and zh signals, which separate paths are shown to simplify conceptualization of the neural net by the reader. In actuality, as previously described, a single path is preferably used to transmit zj in the forward direction and Δj in the reverse direction, etc. Back-propagation processor BPP&sub0; modifies the weights read from word storage elements in neural net layer L&sub0; interstitial memory array by η xi δj amounts and writes them back to the word storage elements in a sequence of read-modify-write cycles during the training procedure. Back-propagation processor BPP&sub1; modifies the weights read from word storage elements in neural net layer L&sub1; interstitial memory array by η xh δi amounts and writes them back to the word storage elements in a sequence of read-modify-write cycles, during the training procedure. Back-propagation processor BPP&sub2; modifies the weights read and storage elements in neural net layer L&sub2; interstitial memory array by η xg δhh amounts and writes them back to the word storage element in a sequence of read-modify-write cycles during the training procedure.

FIGURE 27, comprising component FIGURES 27A and 27B shows an alternative modification that can be manifoldly made to a neural net as shown in FIGURES 2, 8 and 9 or in FIGURES 3, 8 and 9 to give it training capability. This alternative modification seeks to avoid the need for a high-resolution bit-serial multiplier BSM&sub0; and complex addressing during back-propagation calculations in order that training can be implemented. A respective up/down counter UDCi,j is used instead of each word storage element WSEi,j. Correction of the word stored in counter UDCi,j is done a count at a time; and the counter preferably has at least one higher resolution stage in addition to those used to control the capacitances of digital capacitors DCi,j, DC(M+i),j, DCi,(N+j) and DC(M+i),(N+j). Each up/down counter UDCi,j has a respective counter control circuit CONi,j associated therewith- Each counter control circuit CONi,j may, as shown in FIGURE 27A and described in detail further on in this specification, simply consist of'an exclusive-OR gate XORi,j.

Responsive to a SIGN BIT FLAG, a row sign bit latch RBLj latches the sign bit of δj, indicative of whether a row of weights should in general be decremented or incremented, to be applied via a respective row sign line RSLj to all counter control circuits (CONi,j for i=1, ...,M) in the jth row associated with that row sign bit latch RBLj. Before making a back-propagation calculation, a respective column sign bit latch CBLi latches the sign bit of xi for each ith columnar position along the row which is to be updated, to provide an indication of whether it is likely the associated weight should be decremented or incremented. Each column sign bit latch CBLi is connected to broadcast its estimate via a respective column sign line CSLi to all counter control circuits (CONi,j for j=1, ...N) in the ith column associated with that column sign bit latch CBLi. Responsive to these indications from sign bit latches CBLi and RBLj, each respective counter control circuit CONi,j decides in which direction up/down counter UDCi,j will count to adjust the weight control signals Di,j and Di,j stored therein.

The counter control circuitry CONi,j should respond to the sign of +δj being positive, indicating the response vj to be too positive, to decrease the capacitance to output line OLj that is associated with the signal xi or -xi that is positive and to increase the capacitance to output line OLj that is associated with the signal -xi or xi that is negative, for each value of i. The counter control circuitry CONi,j should respond to the sign of +δj being negative, indicating the response v to be too negative, to increase the capacitance to output line OLj that is associated with the signal -xi or xi that is negative and to decrease the capacitance to output line OLj that is associated with the signal xi or -xi that is positive. Accordingly, counter control circuitry CONi,j may simply consist of a respective exclusive-OR gate XORi,j as shown in FIGURE 27A, if the following presumptions are valid.

Each of the digital capacitors DCi,j and DC(M+i),(N+j) is presumed to increase or decrease its capacitance as Di,j is increased or decreased respectively. Each of the digital capacitors DC(M+i),j and DCi,(N+j) is presumed to increase or decrease its capacitance as Di,j is increased or decreased respectively. A ZERO applied as up/down signal to up/down counter UDCi,j is presumed to cause counting down for Di,j and counting up for Di,j. A ONE applied as up/down signal to up/down counter UDCi,j is presumed to cause counting up for Di,j and counting down for Di,j. Column sign detector CSDi output indication is presumed to be a ZERO when xi is not negative and to be a ONE when xi is negative. Row sign detector RSDj output indication is presumed to be a ZERO when δj is not negative and to be a ONE when δj is negative. Since the condition where xi or δj is zero-valued is treated as if the zero-valued number were positive, forcing a false correction which is in fact not necessary, and thus usually creating the need for a counter correction in the next cycle of back-propagation training, there is dither in the correction loops. However, the extra stage or stages of resolution in each up/down counter UDCi,j prevent high-resolution dither in the feedback correction loop affecting the capacitances of DCi,j, DC(M+i),j, DCi,(N+j) and DC(M+i),(N+j).

FIGURE 28, comprising component FIGURES 28A and 28B shows an alternative modification that can be manifoldly made to a neural net as shown in FIGURES 4, 8 and 9 to give it training capability. A respective up/down counter UDCi,j is used instead of each word storage element WSEi,j in this FIGURE 28 alternative modification; and FIGURE 28 differs from FIGURE 21 in substantially the same respects that FIGURE 27 differs from FIGURE 20.

FIGURE 29, comprising component FIGURES 29A and 29B shows an alternative modification that can be manifoldly made to a neural net as shown in FIGURES 12, 18 and 19 or in FIGURES 13, 18 and 19 to give it training capability. A respective up/down counter UDCi,j is used instead of each word storage element WSEi,j in this FIGURE 29 alternative modification; and FIGURE 29 differs from FIGURE 22 in substantially the same respects that FIGURE 27 differs from FIGURE 20. In FIGURE 29 as in FIGURE 22 the signs of the digital signals being weighted and summed are taken into account in the partial weighted summation procedure performed on an analog basis, rather than being taken into account in the final weighted summation procedure performed on an digital basis as is the case in the types of neural nets shown in FIGURES 20 and 27. Accordingly, a neural net manifoldly using the FIGURE 29 modification differs from a neural net manifoldly using the FIGURE 27 modification in that respective straight-through digital bussing replaces the selective complementor RCMPj for each jth row and the selective complementor CCMPi for each ith column. Accordingly, also, in a neural net manifoldly using the FIGURE 29 modification differs from a neural net manifoldly using the FIGURE 27 modification in that each column driver CDi comprises the multiplexers MXi and MX(M+i) connected per FIGURE 12 or per FIGURE 13, rather than the multiplexers MXi and MX(M+i) connected per FIGURE 2 or per FIGURE 3. Except for these differences, a neural net manifoldly using the FIGURE 29 modification resembles, in both structure and operation, a neural net manifoldly using the FIGURE 27 modification.

In a variant of the FIGURE 28 neural net layer not shown in the drawing, the row driver RDj takes either of the forms the column driver CDi can take in FIGURE 29. This provides for taking care of the sign bit in the digital product δi in the partial weighted summation procedure, so it need not be taken care of in the final weighted summation procedure. Accordingly, the selective complementors CCMPi for each ith column are replaced by respective straight-through bussing. The extraction of the sign bit of the digital product δi for application to the row sign line RSLj is done before the row driver RDj, of course.

FIGURE 30, comprising component FIGURES 30A and 30B shows an alternative modification that can be manifoldly made to a neural net using layers as shown in FIGURE 23 to give it training capability. A respective up/down counter UDCi,j is used instead of each word storage element WSEi,j in this FIGURE 30 alternative modification; and FIGURE 30 differs from FIGURE 23 in similar respects as FIGURE 27 differs from FIGURE 20 and as FIGURE 28 differs from FIGURE 21.

FIGURE 31, comprising component FIGURES 31A and 31B shows an alternative modification that can be manifoldly made to a neural net using layers as shown in FIGURE 24 to give it training capability. A respective up/down counter UDCi,j is used instead of each word storage element WSEi,j in this FIGURE 31 alternative modification; and FIGURE 31 differs from FIGURE 24 in similar respects as FIGURE 28 differs from FIGURE 21 and as FIGURE 29 differs from FIGURE 22. Hybrids of the FIGURE 30 and 31 neural net layers are possible, handling the sign bits of the digital input signals as in either one of the FIGURE 30 and 31 neural net layers and handling the sign bits of the digital error signals as in the other of the FIGURE 30 and 31 neural net layers.

FIGURE 32 shows the construction of counter UDCi,j being one that has a plurality of binary counter stages BCS&sub1;, BCS&sub2;, BCS&sub3; that provide increasingly more significant bits of the weight control signal Di,j and of its one's complement Di,j. FIGURE 33 shows the logic within each binary counter stage which is implemented with MOS circuitry that is conventional in the art. FIGURES 32 and 33 make it clear that the opposite directions of counting for Di,j and Di,j can be controlled responsive to a ZERO or ONE up/down control signal in either of two ways, depending on whether Di,j is taken from Q outputs of the flip-flops and Di,j is taken from their Q outputs, as shown, or whether Di,j is taken from the Q outputs of the flip-flops and Di,j is taken from their Q outputs. If the latter choice had been made instead, each counter control circuit CONi,j would have to consist of a respective exclusive-NOR circuit, or alternatively the CSDi and RSDj sign detectors would have to be of opposite logic types, rather than of the same logic type.

While bit-slicing the plural-bit digital synapse samples and processing the bit slices through the same capacitive weighting network is advantageous in that it provides good guarantee that the partial weighted summation results track are scaled in exact powers of two respective to each other, speedier processing can be required which necessitates bit-slicing be done on a spatial-division-multiplexing, rather than time-division multiplexing, basis. Each bit slice is then processed to generate a respective partial weighted summation result in a respective capacitive weighting network similar to the capacitive weighting networks respectively used to generate respective partial weighted summation results for the other bit slices, and a network of digital adders is used instead of an accumulator for generating each final weighted summation result by combining appropriately shifted partial weighted summation results.


Anspruch[en]
  1. A system for performing a plurality of weighted summations, N in number, of a plurality of input signals, M in number; said system CHARACTERIZED IN THAT:

       each of said input signals is a digital signal;

       said system includes apparatus for supplying each word of each of said M input signals one bit at a time on a bit-slice basis; and

       said system includes apparatus for performing said N weighted summations on a bit-slice basis, which apparatus includes a plurality N in number of analog weighted summers identified by respective ones of consecutive ordinal numbers first through Nth, each said analog weighted summer being connected for receiving in parallel said M analog input signals as identified by respective ones of consecutive ordinal numbers first through Mth, and each said analog weighted summer to generate a respective analog output signal identified by the same one of the consecutive ordinal numbers first through Mth as said analog weighted summer generating it.
  2. A system as set forth in Claim 1 for performing N weighted summations of single-bit digital input signals, said system CHARACTERIZED BY said plurality N in number of analog weighted summers identified by respective ones of consecutive ordinal numbers first through Nth together comprising:

       M pairs of input lines, said input lines identified by respective ones of consecutive ordinal numbers first through 2Mth, each pair including input lines identified by ordinal numbers M apart and each pair being identified by the same ordinal number as the lower of the ordinal numbers identifying the input lines in said pair;

       respective circuitry respectively responding to said first through Mth single-bit digital input signals being ONE, for placing a first prescribed voltage level on the input line identified by the corresponding ordinal number and placing a second prescribed voltage level on the input line identified by the ordinal number M higher;

       respective circuitry respectively responding to said first through Mth single-bit digital input signals being ZERO, for placing said second prescribed voltage level on the input line identified by the corresponding ordinal number and placing said first prescribed voltage level on the input line identified by the ordinal number M higher;

       N output lines identified by respective ones of consecutive ordinal numbers first through Nth;

       a plurality of weighting capacitors having respective first plates and second plates, each of said weighting capacitors having its first plate connected to one of said first through 2Mth input lines and having its second plate connected to one of said first through Nth output lines; and

       N charge sensing circuits identified by respective ones of consecutive ordinal numbers first through Nth, each connected for sensing the charge on said output line identified by the same ordinal number it is and responding to generate the respective analog output signal identified by the same ordinal number it is.
  3. A system as set forth in Claim 1 for performing N weighted summations of single-bit digital input signals, said system CHARACTERIZED BY said plurality N in number of analog weighted summers identified by respective ones of consecutive ordinal numbers first through Nth together comprising:

       M pairs of input lines, said input lines identified by respective ones of consecutive ordinal numbers first through 2Mth, each pair including input lines identified by ordinal numbers M apart and each pair being identified by the same ordinal number as the lower of the ordinal numbers identifying the input lines in said pair;

       respective circuitry respectively responding to said first through Mth single-bit digital input signals being ONE, for placing a first prescribed voltage level on the input line identified by the corresponding ordinal number and placing a second prescribed voltage level on the input line identified by the ordinal number M higher;

       respective circuitry respectively responding to said first through Mth single-bit digital input signals being ZERO, for placing a third voltage level both on the input line identified by the corresponding ordinal number and on the input line identified by the ordinal number M higher, said third voltage level being midway between said first and second prescribed voltage levels;

       N output lines identified by respective ones of consecutive ordinal numbers first through Nth;

       a plurality of weighting capacitors having respective first plates and second plates, each of said weighting capacitors having its first plate connected to one of said first through 2Mth input lines and having its second plate connected to one of said first through Nth output lines; and

       N charge sensing circuits identified by respective ones of consecutive ordinal numbers first through Nth, each said charge sensing circuit connected for sensing the charge on said output line identified by the same ordinal number it is and responding to generate the respective analog output signal identified by the same ordinal number it is.
  4. A system as set forth in Claim 1 for performing N weighted summations of single-bit digital input signals, said system CHARACTERIZED BY said plurality N in number of analog weighted summers identified by respective ones of consecutive ordinal numbers first through Nth together comprising:

       N pairs of output lines, said output lines identified by respective ones of consecutive ordinal numbers first through 2Nth, each pair including output lines identified by ordinal numbers N apart and each pair being identified by the same ordinal number as the lower of the ordinal numbers identifying the output lines in said pair;

       M input lines, said input lines identified by respective ones of consecutive ordinal numbers first through Nth;

       respective circuitry respectively responding to said first through Mth single-bit digital input signals being ZERO, for placing a first prescribed voltage level on the input line identified by the corresponding ordinal number;

       respective circuitry respectively responding to said first through Mh single-bit digital input signals being ONE, for placing a second voltage level on the input line identified by the corresponding ordinal number;

       a plurality of weighting capacitors having respective first plates and second plates, each of said weighting capacitors having its first plate connected to one of said first through 2Nth output lines and having its second plate connected to one of said first through Mth input lines;

       N charge sensing circuits identified by respective ones of consecutive ordinal numbers first through Nth, each of said first through Nth charge sensing circuits connected for sensing the differential charge between said output line identified by the same ordinal number it is and said output line identified by the ordinal number N higher, each of said first through Nth charge sensing circuits responding to the differential charge it senses to generate the respective analog output signal identified by the same ordinal number it is; and

       circuitry for periodically zeroing said system during which periods of zeroing said system said first prescribed voltage level is applied to the first and second plates of each of said plurality of weighting capacitors.
  5. A system for performing N weighted summations of input signals as set forth in any of Claims 1-4 CHARACTERIZED BY being included in a neural network layer with:

       N non-linear processors, each having a sigmoidal transfer characteristic, being receptive of a respective one of said first through Nth analog output signals, and responding to generate a respective axonal response.
  6. A system for performing N weighted summations of input signals as set forth in any of Claims 1-4 CHARACTERIZED BY being included in a neural network layer with:

       N non-linear processors, each having a sigmoidal transfer characteristic,being receptive of a respective one of said first through Nth analog output signals, and responding to generate a respective axonal response; and

       analog-to-digital conversion circuitry for digitizing each of said N axonal responses.
  7. A system for performing N weighted summations of input signals as set forth in any of Claims 1-4 CHARACTERIZED BY being included in combination with:

       analog-to-digital conversion circuitry for digitizing said first through Nth analog output signals.
  8. A system for performing N weighted summations of input signals as set forth in Claim 7 CHARACTERIZED BY:

       each of said M digital input signals having B bits in each successive word thereof,

       B being an integer at least two,

       each of said M digital input signals having its words aligned in time with the words of each other of said M digital input signals; and

       a plurality N in number of digital weighted accumulators identified by respective ones of consecutive ordinal numbers first through Nth, each of said digital weighted accumulators being of a type for receiving sequentially via said analog-to-digital conversion circuitry from said analog weighted summer identified by the same ordinal number as its own B digital input signals each representative of a respective partial weighted summation result, which respective partial weighted summation results are shifted in accordance with the weights of the bit slices respectively giving rise to them and then summed to generate a respective digital output signal representative of a respective final weighted summation result.
  9. A system for performing N weighted summations of input signals as set forth in Claim 8 CHARACTERIZED BY being included in a neural network layer with:

       N non-linear digital circuits, each having a sigmoidal transfer characteristic and receptive of a respective one of said digital output signals representative of final weighted summation results, to generate a respective axonal response.
  10. A system for performing N weighted summations of input signals as set forth in any of Claims 1-4 CHARACTERIZED BY being included in a neural network layer with:

       analog-to-digital conversion circuitry for digitizing said first through Nth analog output signals; and

       N non-linear digital circuits, each having a sigmoidal transfer characteristic and receptive of a respective one of said digitized first through Nth analog output signals, to generate a respective axonal response.
  11. A system for performing N weighted summations of input signals as set forth in any of Claims 1-4 CHARACTERIZED BY:

       each of said M digital input signals having B bits in each successive word thereof,

       B being an integer at least two, and

       each of said M digital input signals having its words aligned in time with the words of each other of said M digital input signals.
  12. A system for performing N weighted summations of input signals as set forth in Claim 11 CHARACTERIZED BY being included in combination with:

       a plurality N in number of analog weighted accumulators identified by respective ones of consecutive ordinal numbers first through Nth, each connected for receiving sequentially the B successive values of said analog output signal identified by the same ordinal number it is, as generated responsive to respective bit slices of said M input signals and weighting those B sequentially received values of said analog output signal in accordance with the weights of the bit slices respectively giving rise to them to generate a final weighted summation signal.
  13. A system for performing N weighted summations of input signals as set forth in Claim 12 CHARACTERIZED BY being included in a neural network layer with:

       N non-linear processors, each having a sigmoidal transfer characteristic, being receptive of a respective one of said final weighted summation signals, and responding to generate a respective axonal response.
  14. A system for performing N weighted summations of input signals as set forth in Claim 12 CHARACTERIZED BY being included in a neural network layer with:

       N non-linear processors, each having a sigmoidal transfer characteristic, being receptive of a respective one of said final weighted summation signals, and responding to generate a respective axonal response; and

       analog-to-digital conversion circuitry for digitizing each of said N axonal responses.
  15. A system for performing N weighted summations of input signals as set forth in Claim 12 CHARACTERIZED BY being included in combination with:

       analog-to-digital conversion circuitry for digitizing each of said final weighted summation signals.
  16. A system for performing N weighted summations of input signals as set forth in Claim 12 CHARACTERIZED BY being included in a neural network layer with:

       analog-to-digital conversion circuitry for digitizing each of said final weighted summation signals; and

       N non-linear digital circuits, each having a sigmoidal transfer characteristic and receptive of a respective one of said digitized final weighted summation signals, to generate a respective axonal response.
  17. A system for performing N weighted summations of digital input signals connected in a neural net layer, as set forth in any of Claims 9, 10, 13, 14 and 16, CHARACTERIZED BY said neural net layer being one layer of a plural-layer neural net.






IPC
A Täglicher Lebensbedarf
B Arbeitsverfahren; Transportieren
C Chemie; Hüttenwesen
D Textilien; Papier
E Bauwesen; Erdbohren; Bergbau
F Maschinenbau; Beleuchtung; Heizung; Waffen; Sprengen
G Physik
H Elektrotechnik

Anmelder
Datum

Patentrecherche

Patent Zeichnungen (PDF)

Copyright © 2008 Patent-De Alle Rechte vorbehalten. eMail: info@patent-de.com