The present invention relates to an image processing apparatus, and image processing method, a storage medium, and a program. More particularly, the present invention relates to an image processing apparatus, and image processing method, a storage medium, and a program, adapted to record information indicating a process to be performed on an image in association with a feature value of the image thereby allowing it to reproduce the processed image each time the image is played back.

Many techniques to easily processing images are available.

Most techniques of editing an image taken by a video camera or the like are based on a linear editing method in which when original image data is being played back, various processes are performed on the played-back image and a resultant image is recorded on a storage medium.

However, the linear editing has a problem that when original image data is protected in a copy-once mode, if the original image data is once recorded on a recording apparatus such as a VTR (Video Tape Recorder), it is not allowed to record an image obtained by editing the image played back from the recording apparatus.

To avoid the above problem, there is proposed a technique to record an operation performed to specify a process to edit an image (for example, see Japanese Unexamined Patent Application Publication No. 2003-202944 ).

Referring to Figs. 1 and 2, a known technique of recording and editing an image is described below. In Fig. 1, a horizontal axis indicates "time" of a content recorded, wherein "time" is represented by time code. In Fig. 1, a vertical axis indicates a setting-value recorded and played back. Herein, the setting-value is, for example, a zoom ratio, coordinates of a zoom point, an image parameter such as brightness, contrast, or color, etc.

At the top of Fig. 1, a shaded part of a content indicates a part subjected to editing. In the example shown in Fig. 1, the same part of the content is played back three times or more, and an effect/edit control operation performed on the content is recorded and played back in synchronization with the content.

In the second part immediately below the top part of Fig. 1, a setting-value recorded in a first-time effect/edit control operation is shown. In the first-time effect/edit control operation, as shown in Fig. 1, recording of the setting-value is started when the effect/edit control operation starts, and the recording is ended when the effect/edit control operation ends. Note that at the start point T1 at which recording of the setting-value is started, the setting-value has a normal value. If an operation end button or the like is operated at the end point T2, the effect/edit control operation is ended, and accordingly recording is ended. Thus, in many cases, a discontinuous change in setting-value occurs at the end of the effect/edit control operation, that is, the setting-value returns discontinuously to the normal value at the end of the effect/edit control operation as is the case in the example shown in the second part of Fig. 1.

In the third part, below the second part, of Fig. 1, a setting-value recorded in a second-time effect/edit control operation is shown. In the second-time effect/edit control operation, in this example, the part of the content subjected to the process (from the effect/edit control operation start point T1 to the effect/edit control operation end point T2) is partially modified. More specifically, an operation for modification is started at a modification start point T3 and is ended at a modification end point T4. That is, in the second-time effect/edit control operation, when the same content as that subjected to the first-time effect/edit control operation is being played back, an effect/edit control operation different from that performed in the first-time effect/edit control operation is performed from the modification start point T3 to the modification end point T4 as shown in the third part of Fig. 1.

In the third part of Fig. 1, a broken line indicates the setting-value recorded in the first-time effect/edit control operation.

After the first-time effect/edit control operation is performed (that is, after the setting-value is recorded in the first-time effect/edit control operation), a user has to rewind the content to the effect/edit control operation start point T1 before the second-time effect/edit control operation is started. After the setting-value is recorded in the second-time effect/edit control operation, if the content is again rewound and played back, the setting-value is reproduced in such a manner as shown in the fourth part of Fig. 1. Note that in the second-time effect/edit control operation, the setting-value is overwritten on the setting-value recorded in the first-time effect/edit control operation. Thus, in many cases, a discontinuous change occurs in the setting-value at the end of the overwritten part.

Referring to Figs. 2A to 2C, a specific example of an effect/edit control operation is described below. Fig. 2A shows a first-time effect/edit control operation and a setting-value recorded in the first-time effect/edit control operation. Fig. 2B shows a second-time effect/edit control operation and a setting-value recorded in the second-time effect/edit control operation. Fig. 2C shows a setting-value recorded as an overall result of the first and second effect/edit control operations. In each of Figs. 2A and 2B, an upper part shows a setting-value, and a lower part shows an effect/edit control operation.

In the example shown in Fig. 2A, the first-time effect/edit control operation is performed such that an operation signal such as that shown in the lower part of Fig. 2A is output from a controller in accordance with an operation performed on the controller by a user, and a corresponding setting-value is recorded as shown in the upper part of Fig. 2A. Note that the setting-value is given by the integral of the amount of the operation performed on the controller. In Figs. 2A and 2B, a "positive operation" refers to a positive effect/edit control operation such as enlarging of an image, and a "negative operation" refers to a negative effect/edit control operation such as reducing of an image.

In the example shown in Fig. 2A, a positive effect/edit control operation is performed in a period from an effect/edit control operation start point T1 to an effect/edit control operation end point T11 (see the lower part of Fig. 2A), and the integral of the amount of the operation is recorded as the setting-value. Thus, the recorded setting-value gradually increases as shown in the upper part of Fig. 2A. In a period from a point of time T11 to a point of time T12, a negative effect/edit control operation is performed, and thus the resultant setting-value gradually decreases. After the point of time T12, a positive effect/edit control operation and a negative effect/edit control operation are performed, and a setting-value is recorded in accordance with the operations.

If an operation end button (a record end button) on the controller is operated at the point of time T2, recording of the setting-value is ended. When the recording is ended, the setting-value is returned to the normal value. As a result, the setting-value is recorded as shown in the upper part of Fig. 2A.

As shown in the lower part of Fig. 2B, if a second-time effect/edit control operation is performed on the content subjected to the first-time effect/edit control operation such that a positive effect/edit control operation is performed in a period from a time T3 to a time T21 and a negative effect/edit control operation is performed in a period from the time T21 to a time T4, then a setting-value is recorded in the period from the time T3 to the time T4 as shown in the upper part of Fig. 2B.

In the second-time effect/edit control operation, the integral of the amount of the effect/edit control operation is added to the setting-value recorded at the point of time T3 in the first-time effect/edit control operation. When the effect/edit control operation is ended at the point of time T4, the setting-value is returned to the setting-value recorded at the point of time T4 in the first-time effect/edit control operation.

A setting-value produced as a result of the solidification is recorded as represented by a solid line in the upper part of Fig. 2B (that is, the setting-value is modified to the value represented by the solid line). Thus, after the second-time effect/edit control operation, the overall setting-value is recorded as shown in Fig. 2C.

In the example shown in Fig. 2C, an abrupt reduction occurs in the setting-value at the point of time T4 and another abrupt reduction occurs at the point of time T2. In the case where the setting-value specifies, for example, a zoom ratio of an image, such an abrupt change in the setting-value causes the image to be abruptly enlarged or reduced. Such an abrupt change in image gives a discomfort impression to a user.

Furthermore, as can be seen from Fig. 2C, the setting-value always starts from the normal value and ends at the normal value. This makes it difficult to freely specify the setting-value at the record start point and the record end point, and there is a possibility that the setting-value is discontinuously returned to the normal value, which can cause an abrupt change in the image as described above.

In a case where editing is performed a plurality of times (two times in the example described above with reference to Figs. 2A to 2C), a setting-value designated in a second-time edit operation is overwritten on a setting-value recorded in the previous operation, and a discontinuity of the setting-value often occurs at the end of the overwriting (at T4 in the above-described example), which can cause the above-described problem.

That is, in the known recording method, the effect/edit control operation is directly recorded, and no correction is made to obtain a smooth change in setting-value. Thus, it is difficult to avoid an abrupt change in an image.

Another problem of overwriting is in that if editing such as compensation for camera shake is performed on an automatically recorded content, then further editing is not allowed for the result.

In view of the above, it is desirable to provide a technique to edit an image without causing an abrupt change.

According to an embodiment of the present invention, there is provided a first image processing apparatus including feature value extraction means for extracting a feature value from an image, setting means for setting setting-information defining a process to be performed on the image, edit means for editing the setting-information set by the setting means, and recording means for recording the setting-information edited by the edit means, in association with the feature value of the image, on a data recording medium.

In this first image processing apparatus, when the setting by the setting means is ended, the edit means may edit the setting-information between first setting-information given at the end of the setting by the setting means and second setting-information defined as normal setting-information such that the setting-information gradually changes from the first setting-information to the second setting-information.

In the first image processing apparatus, when the setting by the setting means is ended, the edit means may edit the setting-information between first setting-information given at the end of the setting by the setting means and second setting-information which has been assigned for the image at a point of time later than the point of time corresponding to the first setting-information and which has been recorded on the data recording medium such that the setting-information gradually changes from the first setting-information to the second setting-information.

In the first image processing apparatus, the edit means may subtract or add a predetermined value from or to a value indicated by the first setting-information, and may assign third setting-information indicating the result of the subtraction or the addition to the image at a point of time next to the point of time corresponding to the first setting-information, if the value indicated by the third setting-information is not within a predetermined range from the value indicated by the second setting-information, the edit means may further subtract or add the predetermined value from or to the value indicated by the third setting-information, and may assign setting-information, as new third setting-information, indicating the result of the subtraction or the addition to the image at a further next point of time, and if the value indicated by the third setting-information falls within the predetermined range from the value indicated by the second setting-information, the edit means may employ the third setting-information as the second setting-information.

In the first image processing apparatus, the edit means may add a value indicated by setting-information set by the setting means to a value indicated by setting-information which has been set for the same image as the image for which the former setting-information is assigned and which has been recorded on the data recording medium.

The first image processing apparatus may further include key point setting means for setting a key point specifying a start point or an end point of a period during which editing is to be performed by the edit means, wherein the edit means may edit the setting-information such that the setting-information gradually changes in value between two adjacent key points of the key points set by the key point setting means.

The key point setting means may set the key points at a point of time before recording is performed by the recording means or at a point of time after recording is performed by the recording means.

According to another embodiment of the present invention, there is provided a first image processing method including steps of extracting a feature value from an image, setting setting-information defining a process to be performed on the image, editing the setting-information set in the setting step, and controlling a recording operation so as to record the setting-information edited in the editing step, in association with the feature value of the image, on a data recording medium.

According to another embodiment of the present invention, there is provided a first storage medium on which a program is stored, the program being executable by a computer to perform a process including the steps of extracting a feature value from an image, setting setting-information defining a process to be performed on the image, editing the setting-information set in the setting step, and controlling a recording operation so as to record the setting-information edited in the editing step, in association with the feature value of the image, on a data recording medium.

According to another embodiment of the present invention, there is provided a first program executable by a computer to perform a process including the steps of extracting a feature value from an image, setting setting-information defining a process to be performed on the image, editing the setting-information set in the setting step, and controlling a recording operation so as to record the setting-information edited in the editing step, in association with the feature value of the image, on a data recording medium.

In the first image processing apparatus, the first image processing method, and the first program, a feature value is extracted from an image, setting-information is set to define a process to be performed on the image, the setting-information is edited, and the resultant setting-information is recorded in association with the feature value of the image on a data recording medium.

According to an embodiment of the present invention, there is provided a second image processing apparatus including feature value extraction means for extracting a feature value from an image, setting means for setting setting-information defining a process to be performed on the image, recording means for recording the setting-information, in association with the feature value of the image, on a data recording medium, and control means for determining whether the recording means should record the setting-information on the data recording medium, and controlling recording performed by the recording means in accordance with a determination result.

The control means may determine that the recording means should not record the setting-information on the data recording medium, when at least one of the following conditions is satisfied: (a) identical setting-information is set continuously by the setting means, (b) the setting-information set by the setting means is close to corresponding setting-information recorded on the data recording medium, and (c) the setting-information set by the setting means is close to setting-information defined as normal setting-information.

The second image processing apparatus may further include specifying means for specifying a start point at which to start recording by the recording means and an end point at which to end the recording, wherein the control means may determine that recording of the setting-information on the data recording medium by the recording means is to be started at the start point specified by the specifying means, and the control means may determine that recording of the setting-information on the data recording medium by the recording means is to be ended at the end point specified by the specifying means.

The specifying means may specify the start point and the end point at a point of time before recording is performed by the recording means.

The recording means may temporarily record the setting-information on the data recording medium, the specifying means may specify the start point and the end point after recording is performed by the recording means, and the control means may re-record the setting-information temporarily recorded on the data recording medium such that re-recording of the setting-information is started at the start point and ended at the end point.

The second image processing apparatus may further include detection means for detecting a scene change, wherein when a scene change is detected by the detection means, if recording of setting-information on the data recording medium by the recording means is not being performed, the control means may determine that recording should be started, but if recording of setting-information on the data recording medium by the recording means is being performed, the control means may determine that recording should be ended.

According to an embodiment of the present invention, there is provided a second image processing method including the steps of extracting a feature value from an image, setting setting-information defining a process to be performed on the image, and controlling recording of the setting-information, in association with the feature value of the image, on a data recording medium such that a determination is made as to whether the setting-information should be recorded on the data recording medium, and recording is controlled in accordance with a determination result.

According to an embodiment of the present invention, there is provided a second storage medium on which a program is stored, the program being executable by a computer to perform a process including the steps of extracting a feature value from an image, setting setting-information defining a process to be performed on the image, controlling recording of the setting-information, in association with the feature value of the image, on a data recording medium such that a determination is made as to whether the setting-information should be recorded on the data recording medium, and recording is controlled in accordance with a determination result.

According to an embodiment of the present invention, there is provided a second program executable by a computer to perform a process including the steps of extracting a feature value from an image, setting setting-information defining a process to be performed on the image, controlling recording of the setting-information, in association with the feature value of the image, on a data recording medium such that a determination is made as to whether the setting-information should be recorded on the data recording medium, and recording is controlled in accordance with a determination result.

In the second image processing apparatus, the second image processing method, and the second program, a feature value is extracted from an image, setting-information is set to define a process to be performed on the image, the setting-information is recorded in association with the feature value of the image on a data recording medium, and a determination is made as to whether the setting-information should be recorded on the data recording medium, and recording is controlled in accordance with a determination result.

According to an embodiment of the present invention, there is provided a third image processing apparatus including feature value extraction means for extracting a feature value from an image, operation means for performing an operation to specify a process to be performed on the image, setting means for, in accordance with the operation performed by the operation means, setting setting-information according to which to perform the process on the image, edit means for editing the operation information, and recording means for recording the operation information edited by the edit means, in association with the feature value of the image, on a data recording medium.

In this third image processing apparatus, when the operation by the operation means is ended, the edit means may edit the operation information between first operation information given at the end of the operation by the operation means and second operation information defined as normal operation information such that the operation information gradually changes from the first operation information to the second operation information.

In the third image processing apparatus, when the operation by the operation means is ended, the edit means may edit the operation information between first operation information given at the end of the operation by the operation means and second operation information which has been assigned at a point of time later than the point of time corresponding to the first operation information and which has been recorded on the data recording medium such that the operation information gradually changes from the first operation information to the second operation information.

The edit means may subtract or add a predetermined value from or to a value indicated by the first operation information, and may assign third operation information indicating the result of the subtraction or the addition to the image at a point of time next to the point of time corresponding to the first operation information, if the value indicated by the third operation information is not within a predetermined range from the value indicated by the second operation information, the edit means may further subtract or add the predetermined value from or to the value indicated by the third operation information, and may assign operation information, as new third operation information, indicating the result of the subtraction or the addition to the image at a further next point of time, and if the value indicated by the third operation information falls within the predetermined range from the value indicated by the second operation information, the edit means may employ the third operation information as the second operation information.

The edit means may add a value indicated by the operation information to a value indicated operation information which has been assigned for the same image as the image for which the former operation information is assigned and which has been recorded on the data recording medium.

The third image processing apparatus may further include key point setting means for setting a key point specifying a start point or an end point of a period during which editing is to be performed by the edit means, wherein the edit means may edit the operation information such that the operation information gradually changes in value between two adjacent key points of the key points set by the key point setting means.

The key point setting means may set the key points at a point of time before recording is performed by the recording means or at a point of time after recording is performed by the recording means.

According to an embodiment of the present invention, there is provided a third image processing method including the steps of extracting a feature value from an image, performing an operation to specify a process to be performed on the image, in accordance with the operation performed in the operation step, setting setting-information according to which to perform the process on the image, editing the operation information, and controlling a recording operation so as to record the operation information edited in the editing step, in association with the feature value of the image, on a data recording medium.

According to an embodiment of the present invention, there is provided a third storage medium on which a program is stored, the program being executable by a computer to perform a process including the steps of extracting a feature value from an image, performing an operation to specify a process to be performed on the image, in accordance with the operation performed in the operation step, setting setting-information according to which to perform the process on the image, editing the operation information, and controlling a recording operation so as to record the operation information edited in the editing step, in association with the feature value of the image, on a data recording medium.

According to another embodiment of the present invention, there is provided a third program executable by a computer to perform a process including the steps of extracting a feature value from an image, performing an operation to specify a process to be performed on the image, in accordance with the operation performed in the operation step, setting setting-information according to which to perform the process on the image, editing the operation information, and controlling a recording operation so as to record the operation information edited in the editing step, in association with the feature value of the image, on a data recording medium.

In the third image processing apparatus, the third image processing method, and the third program, a feature value is extracted from an image, an operation is performed to specify a process to be performed on the image, in accordance with the performed operation, setting-information is set according to which to perform the process on the image, operation information is edited, and the resultant operation information is recorded in association with the feature value of the image on a data recording medium.

According to an embodiment of the present invention, there is provided a fourth image processing apparatus including feature value extraction means for extracting a feature value from an image, operation means for performing an operation to specify a process to be performed on the image, setting means for, in accordance with the operation performed by the operation means, setting setting-information according to which to perform the process on the image, recording means for recording operation information indicating the operation performed by the operation means, in association with the feature value of the image, on a data recording medium, and control means for determining whether the recording means should record the operation information on the data recording medium, and controlling recording performed by the recording means in accordance with a determination result.

The control means may determine that the recording means should not record the operation information on the data recording medium, when at least one of the following conditions is satisfied: (d) no signal is supplied from the operation means over a period with a length equal to or greater than a predetermined value, (e) the operation information is close to corresponding operation information recorded on the data recording medium, and (f) the operation information is close to operation information defined as normal operation information.

The fourth image processing apparatus may further include specifying means for specifying a start point at which to start recording by the recording means and an end point at which to end the recording, wherein the control means may determine that recording of the operation information on the data recording medium by the recording means is to be started at the start point specified by the specifying means, and the control means may determine that recording of the operation information on the data recording medium by the recording means is to be ended at the end point specified by the specifying means.

The specifying means may specify the start point and the end point at a point of time before recording is performed by the recording means.

The recording means may temporarily record the operation information on the data recording medium, the specifying means may specify the start point and the end point after recording is performed by the recording means, and the control means may re-record the operation information temporarily recorded on the data recording medium such that re-recording of the operation information is started at the start point and ended at the end point.

The fourth image processing apparatus may further include detection means for detecting a scene change, wherein when a scene change is detected by the detection means, if recording of operation information on the data recording medium by the recording means is not being performed, the control means may determine that recording should be started, but if recording of operation information on the data recording medium by the recording means is being performed, the control means may determine that recording should be ended.

According to an embodiment of the present invention, there is provided a fourth image processing method including the steps of extracting a feature value from an image, performing an operation to specify a process to be performed on the image, in accordance with the operation performed in the operation step, setting setting-information according to which to perform the process on the image, and determining whether the operation information should be recorded on the data recording medium, and controlling recording in accordance with a determination result.

According to an embodiment of the present invention, there is provided a fourth storage medium on which a program is stored, the program being executable by a computer to perform a process including the steps of extracting a feature value from an image, performing an operation to specify a process to be performed on the image, in accordance with the operation performed in the operation step, setting setting-information according to which to perform the process on the image, and determining whether the operation information should be recorded on the data recording medium, and controlling recording in accordance with a determination result.

According to another embodiment of the present invention, there is provided a fourth program executable by a computer to perform a process including the steps of extracting a feature value from an image, performing an operation to specify a process to be performed on the image, in accordance with the operation performed in the operation step, setting setting-information according to which to perform the process on the image, and determining whether the operation information should be recorded on the data recording medium, and controlling recording in accordance with a determination result.

In the fourth image processing apparatus, the fourth image processing method, and the fourth program, a feature value is extracted from an image, an operation is performed to specify a process to be performed on the image, in accordance with the performed operation, setting-information is set according to which to perform the process on the image, the operation information is recorded in association with the feature value of the image on a data recording medium, a determination is made as to whether the operation information should be recorded on the data recording medium, and recording is controlled in accordance with a determination result.

As described above, the present invention provides a great advantage that the setting-value specifying an effect to be applied to an image can be controlled such that no abrupt change in the image occurs, and the resultant setting-value can be recorded.

• Fig. 1 shows a change in a setting-value associated with editing according to a conventional technique;
• Fig. 2 shows a change in a setting-value associated with editing according to a conventional technique;
• Fig. 3 shows a concept of an image processing apparatus according to an embodiment of the present invention;
• Fig. 4 shows a concept of an image processing apparatus according to an embodiment of the present invention;
• Fig. 5 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 6 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 7 is a block diagram showing an example of a configuration of a feature value extractor according to an embodiment of the present invention;
• Fig. 8 is a diagram showing an example of a format of an image signal;
• Fig. 9 is a block diagram showing an example of a configuration of a synchronization detector according to an embodiment of the present invention;
• Fig. 10 is a flow chart showing image processing performed by an image processing apparatus according to an embodiment of the present invention;
• Fig. 11 is a flow chart showing a feature value extraction process according to an embodiment of the present invention;
• Fig. 12 is a flow chart showing a process associated with synchronization according to an embodiment of the present invention;
• Fig. 13 is a flow chart showing a synchronization detection process according to an embodiment of the present invention;
• Fig. 14 is a flow chart showing an effect/edit process according to an embodiment of the present invention;
• Fig. 15 is a diagram illustrating a synchronization detection process according to an embodiment of the present invention;
• Fig. 16 is a diagram illustrating a feature value extraction process according to an embodiment of the present invention;
• Fig. 17 is a diagram illustrating a change in a setting-value associated with a smooth end process according to an embodiment of the present invention;
• Fig. 18 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 19 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 20 is a flow chart showing image processing according to an embodiment of the present invention;
• Fig. 21 is a flow chart showing a process associated with synchronization according to an embodiment of the present invention;
• Fig. 22 is a flow chart showing a smooth end process according to an embodiment of the present invention;
• Fig. 23 is a flow chart showing an effect/edit process according to an embodiment of the present invention;
• Fig. 24 is a diagram illustrating a smooth end process according to an embodiment of the present invention;
• Fig. 25 is a diagram illustrating a change in a setting-value associated with an automatic end process according to an embodiment of the present invention;
• Fig. 26 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 27 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 28 is a flow chart showing image processing according to an embodiment of the present invention;
• Fig. 29 is a flow chart showing a process associated with synchronization according to an embodiment of the present invention;
• Fig. 30 is a flow chart showing an effect/edit process according to an embodiment of the present invention;
• Fig. 31 is a diagram illustrating an addition process according to an embodiment of the present invention;
• Fig. 32 is a diagram illustrating an addition process according to an embodiment of the present invention;
• Fig. 33 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 34 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 35 is a flow chart showing an effect/edit process according to an embodiment of the present invention;
• Fig. 36 is a diagram illustrating designation of a recording range according to an embodiment of the present invention;
• Fig. 37 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 38 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 39 is a flow chart showing a process of specifying a recording range according to an embodiment of the present invention;
• Fig. 40 is a flow chart showing image processing according to an embodiment of the present invention;
• Fig. 41 is a flow chart showing a triggering process according to an embodiment of the present invention;
• Fig. 42 is a flow chart showing an effect/edit process according to an embodiment of the present invention;
• Fig. 43 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 44 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 45 is a flow chart showing a process of specifying a recording range according to an embodiment of the present invention;
• Fig. 46 is a diagram illustrating scene change detection according to an embodiment of the present invention;
• Fig. 47 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 48 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 49 is a flow chart showing image processing according to an embodiment of the present invention;
• Fig. 50 is a flow chart showing a process associated with scene change detection according to an embodiment of the present invention;
• Fig. 51 is a diagram illustrating key points according to an embodiment of the present invention;
• Fig. 52 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 53 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention;
• Fig. 54 is a flow chart showing a key point recording process according to an embodiment of the present invention;
• Fig. 55 is a flow chart showing image processing according to an embodiment of the present invention;
• Fig. 56 is a flow chart showing a key point recording process according to an embodiment of the present invention; and
• Fig. 57 is a diagram illustrating a storage medium and a personal computer on which image processing functions are implemented by software according to an embodiment of the present invention.

Before embodiments of the present invention are described, correspondence between specific examples of parts/steps in the embodiments and those essential to the present invention. Note that the purpose of the following description is to indicate that specific examples of the present invention are described in the embodiments, and thus the purpose of the following description is not to limit the present invention to those described below. That is, there can be a part/step that is not described in the following description of the correspondence but that corresponds to a part/step according to the present invention. Conversely, an example of a part/step, which is described in the following description as corresponding to a particular aspect of the invention, can correspond to a part/step of another aspect of the invention.

According to an embodiment of the present invention, there is provided a first image processing apparatus (for example, an image processing apparatus 13 shown in Fig. 18) including feature value extraction means (for example, a feature value extractor 41 shown in Fig. 18) for extracting a feature value from an image, setting means (for example, a setting-information determination unit 46 shown in Fig. 18) for setting setting-information defining a process to be performed on the image, edit means (for example, a smooth end controller 201 shown in Fig. 18) for editing the setting-information set by the setting means, and recording means (for example, a storage block 43 shown in Fig. 18) for recording the setting-information edited by the edit means, in association with the feature value of the image, on a data recording medium.

According to an embodiment of the present invention, there is provided a second image processing apparatus (for example, an image processing apparatus 13 shown in Fig. 37) including feature value extraction means (for example, a feature value extractor 41 shown in Fig. 37) for extracting a feature value from an image, setting means (for example, a setting-information determination unit 46 shown in Fig. 37) for setting setting-information defining a process to be performed on the image, recording means (for example, a storage block 43 shown in Fig. 37) for recording the setting-information, in association with the feature value of the image, on a data recording medium, and control means (for example, a trigger unit 503 shown in Fig. 37) for determining whether the recording means should record the setting-information on the data recording medium, and controlling recording performed by the recording means in accordance with a determination result.

According to an embodiment of the present invention, there is provided a third image processing apparatus (for example, an image processing apparatus 13 shown in Fig. 19) including feature value extraction means (for example, a feature value extractor 41 shown in Fig. 19) for extracting a feature value from an image, operation means (for example, an operation information analysis unit 45 shown in Fig. 19) for performing an operation to specify a process to be performed on the image, setting means (for example, a setting-information determination unit 153 shown in Fig. 19) for, in accordance with the operation performed by the operation means, setting setting-information according to which to perform the process on the image, edit means (for example, a smooth end controller 211 shown in Fig. 19) for editing the operation information, and recording means (for example, a storage block 151 shown in Fig. 19) for recording the operation information edited by the edit means, in association with the feature value of the image, on a data recording medium.

According to an embodiment of the present invention, there is provided a fourth image processing apparatus (for example, an image processing apparatus 13 shown in Fig. 38) including feature value extraction means (for example, a feature value extractor 41 shown in Fig. 38) for extracting a feature value from an image, operation means (for example, an operation information analysis unit 45 shown in Fig. 38) for performing an operation to specify a process to be performed on the image, setting means (for example, a setting-information determination unit 153 shown in Fig. 38) for setting setting-information defining a process to be performed on the image, recording means (for example, a storage block 151 shown in Fig. 38) for recording operation information indicating the operation performed by the operation means, in association with the feature value of the image, on a data recording medium, and control means (for example, a trigger unit 513 shown in Fig. 38) for determining whether the recording means should record the operation information on the data recording medium, and controlling recording performed by the recording means in accordance with a determination result.

Now, embodiments of the present invention are described below with reference to the accompanying drawings.

First, referring to Figs. 3 and 4, a concept of an image processing apparatus according to an embodiment of the present invention is described below. In the present embodiment, the image processing apparatus is adapted to perform a process specified by a user on an image recorded in advance. In this process, the image processing apparatus extracts a feature value of the image and records information indicating the process specified by the user in association with the feature value. When the image recorded in advance is played back, the image processing apparatus reads the recorded information indicating the process to be performed on the image and processes the image in accordance with the read information.

More specifically, the image processing apparatus 2 has two operation modes: a recording mode, and a playback mode.

In the recording mode, as shown in Fig. 3, the image processing apparatus 2 reads a content including a moving image recorded in advance on a recording medium 1 such as a DVD (Digital Versatile Disc) via a DVD player or the like, and the image processing apparatus 2 displays the image on a display unit 3. When the image is being displayed, if a user operates a remote control or the like to perform an effect/edit process such as zooming, panning, or tilting on the image, the image processing apparatus 2 generates setting-information corresponding to the operation performed on the remote control (hereinafter referred to as an effect/edit control operation), performs the effect/edit process corresponding to the setting-information on the image, and displays the resultant image on the display unit 3. Furthermore, the image processing apparatus 2 extracts a feature value from the image and records the setting-information, in association with the extracted feature value, in an operation log data recording unit 4.

On the other hand, in the playback mode, as shown in Fig. 4, the image processing apparatus 2 reads a content including a moving image played back by a DVD player or the like from a recording medium 1 and extracts a feature value of the image. In this playback operation, the image processing apparatus 2 reads the setting-information associated with the feature value from the operation log data recording unit 4 in synchronization with the feature value of the image read from the recording medium 1, performs the process on the image in accordance with the read setting-information, and displays the resultant image on the display unit 3.

Thus, the image processing apparatus 2 records only the information (setting-information) indicating the process performed on the pre-recorded image without recording the image obtained as a result of the process so that the image is processed in accordance with the recorded setting-information and the resultant image is output in the playback operation.

The image processing apparatus 2 shown in Figs. 3 and 4 are described in further detail below.

Fig. 5 shows an example of a configuration of an image processing apparatus 13, which corresponds to the image processing apparatus 2 shown in Fig. 3 or 4, according to an embodiment of the present invention.

An image playback unit 12 reads image data recorded in an encoded form on a recording medium 11 corresponding to the recording medium 1 shown in Fig. 3 or 4, and the image playback unit 12 decodes the image data. The resultant decoded image data is supplied to a feature value extractor 41 and a delay unit 48.

The feature value extractor 41 extracts a feature value of each frame of the image data supplied from the image playback unit 12 and supplies the extracted feature value to the synchronization detector 42 and a feature value/setting-information recording control unit 61 in a storage block 43. The detailed configuration of the feature value extractor 41 will be described later with reference to Fig. 7.

A remote control 14 including keys and buttons is used by a user to designate a process to be performed on the image, as shown in Fig. 3 or 4. If the remote control 14 is operated by the user, the remote control 14 generates an operation signal corresponding to the operation performed by the user and emits an infrared ray or the like having an emission pattern corresponding to the operation signal to a light receiver 44 of the image processing apparatus 13.

If the light receiver 44 receives the infrared ray from the remote control 14, the light receiver 44 converts the optical signal of the infrared ray into the original operation signal indicating the operation performed on the remote control 14 and supplies the resultant operation signal to an operation information analysis unit 45. The operation information analysis unit 45 analyzes the received operation signal supplied from the light receiver 44 to detect operation information corresponding to a process specified, by the user, to be performed on the image. The resultant operation information is supplied to a setting-information determination unit 46.

In accordance with the operation information supplied from the operation information analysis unit 45, the setting-information determination unit 46 determines setting-information indicating the process to be performed on the image by an applying unit 49, which will be described in detail later, and the setting-information determination unit 46 supplies the determined setting-information to the feature value/setting-information recording control unit 61 and a selector 47.

The feature value/setting-information recording control unit 61 in the storage block 43 records the feature value supplied from the feature value extractor 41 and the setting-information supplied from the setting-information determination unit 46 in a feature value/setting-information recording unit 62 (corresponding to the operation log data recording unit 4 shown in Fig. 3 or 4) in such a manner that the feature value and the setting-information are associated with each other.

The synchronization detector 42 searches the feature value/setting-information recording unit 62 to detect a feature value (a retrieved-frame feature value described later) identical to the feature value extracted, by the feature value extractor 41, from the image being played back by the image playback unit 12. If the identical feature value is detected, the synchronization detector 42 supplies the detected feature value indicating the synchronous image position to a feature value/setting-information reading unit 63. The detailed configuration of the synchronization detector 42 will be described later with reference to Fig. 9.

If the feature value/setting-information reading unit 63 acquires the feature value (the retrieved-frame feature value) indicating the synchronous position from the synchronization detector 42, the feature value/setting-information reading unit 63 reads setting-information corresponding to the acquired feature value from the feature value/setting-information recording unit 62 and supplies the read setting-information to the selector 47. If the selector 47 receives the setting-information from the setting-information determination unit 46, the selector 47 supplies this setting-information to the applying unit 49 regardless of whether setting-information is supplied from the feature value/setting-information reading unit 63 at the same time as the time at which the setting-information is supplied from the setting-information determination unit 46. On the other hand, in a case where no setting-information is supplied from the setting-information determination unit 46 but setting-information is supplied from the feature value/setting-information reading unit 63, the selector 47 transfers the setting-information supplied from the feature value/setting-information reading unit 63 to the applying unit 49. In a case where setting-information is supplied from neither, the selector 47 supplies no setting-information to the applying unit 49.

The delay unit 48 temporarily stores the image data supplied from the image playback unit 12 for a period equal to a total delay time caused by processes performed by the feature value extractor 41, the synchronization detector 42, the storage block 43, and the selector 47, and the delay unit 48 outputs the resultant delayed image data to the applying unit 49. In the case where the applying unit 49 receives setting-information from the selector 47, the applying unit 49 applies the specified process to the image supplied from the delay unit 48 and displays the resultant image on the display unit 15. On the other hand, when no setting-information is supplied from the selector 47 to the applying unit 49, the applying unit 49 directly displays the image supplied from the delay unit 48 on the display unit 15.

Fig. 6 shows another example of the configuration of the image processing apparatus 13. In this example shown in Fig. 6, the image processing apparatus 13 is configured to record not setting-information but operation information in association with the feature value. In Fig. 6, similar parts to those in Fig. 5 are denoted by similar reference numerals, and a duplicated explanation thereof is omitted herein.

That is, a feature value extractor 41, a synchronization detector 42, a light receiver 44, an operation information analysis unit 45, a delay unit 48, and an applying unit 49 are similar to those in the image processing apparatus 13 shown in Fig. 5. However, the image processing apparatus 13 shown in Fig. 6 is different from the image processing apparatus 13 shown in Fig. 5 in that the storage block 43, the setting-information determination unit 46, and the selector 47 are respectively replaced with a storage block 151, a setting-information determination unit 153, and a selector 152.

The storage block 151 stores the operation information supplied from the operation information analysis unit 45 in association with the feature value. The storage block 151 also reads stored operation information corresponding to a feature value of an image being currently played back, and supplies the read operation information to the selector 152.

The feature value/operation information recording control unit 161 in the storage block 151 stores the feature value supplied from the feature value extractor 41 and the operation information supplied from the operation information analysis unit 45, in association with each other, in the feature value/operation information recording unit 162.

If the feature value/operation information reading unit 163 receives a retrieved-frame feature value, that is, a feature value indicating a synchronous image position, from the synchronization detector 42, the feature value/operation information reading unit 163 reads operation information stored in association with the received feature value and supplies it to the selector 152. In the case where the selector 152 receives the operation information from the operation information analysis unit 45, the selector 152 transfers the operation information supplied from the operation information analysis unit 45 to the setting-information determination unit 153 regardless of whether operation information is received from the feature value/operation information reading unit 163. On the other hand, in the case where the selector 152 receives no operation information from the operation information analysis unit 45 but the selector 152 receives operation information from the feature value/operation information reading unit 163, the selector 152 transfers the operation information supplied from the feature value/operation information reading unit 163 to the setting-information determination unit 153. In a case where operation information is supplied from neither, the selector 152 supplies no operation information to the setting-information determination unit 153.

In accordance with the operation information supplied from the selector 152, the setting-information determination unit 153 determines setting-information indicating the process to be performed on the image by the applying unit 49 and supplies it to the applying unit 49.

In the image processing apparatus 13 shown in Fig. 6, the feature value extractor 41 and the synchronization detector 42 are similar to those used in the image processing apparatus 13 shown in Fig. 5. The feature value extractor 41 and the synchronization detector 42 are described in detail below.

First, referring to Fig. 7, an example of a configuration of the feature value extractor 41 is described in detail.

A DFF (D-flip-flop) 81 stores an immediately previous input signal and outputs it to an adder 82 in synchronization with a clock signal (CLK) supplied from a clock signal generator (not shown). When the input signal is out of an image data area, a reset signal is input to the DFF 81 such that the input signal is deleted and no image signal is output. Note that, as shown in Fig. 8, the image signal includes a synchronization data area and an image data area. Application of the reset signal to the DFF 81 is controlled in accordance with position information indicating the position of the image signal with respect to the frame start position S, which is the intersection of the horizontal axis (sample) and the vertical axis (line). If the position information indicates that the image signal is in the synchronization data area and is output of the image data area, the reset signal is applied to the DFF 81 and thus the image signal is output to the adder 82. That is, only the image signal in the image data area is supplied from the DFF 81 to the adder 82 in synchronization with the clock signal.

The adder 82 adds the signal supplied from the DFF 81 and a signal supplied from a DFF 83, and supplies a resultant signal to the DFF 83. More specifically, the adder 82 extracts low-order 8 bits of the sum of the signal supplied from the DFF 81 and the signal supplied from the DFF 83, and supplies the extracted 8 bits to the DFF 83.

The signal input to the DFF 83 from the adder 82 is output to the adder 82 and an output unit 84 in synchronization with the clock signal (CLK) supplied from the clock generator (not shown). When the signal input to the DFF 83 is a signal at the frame start point (the intersection of the horizontal axis (sample) and the vertical axis (line)), the reset signal is input to the DFF 83 such that the input signal is deleted and no signal is output from the DFF 83. That is, the input signal is added cumulatively over the image data area by the adder 82 in cooperation with the DFF 83, and the resultant cumulative sum is output from the DFF 83 to the output unit 84.

When the output unit 84 receives the cumulative sum for one frame (or one field) from the DFF 83, the output unit 84 outputs the cumulative sum as the feature value of the one frame of image. Note that the value output as the feature value of the one frame (or field) of image from the output unit 84 is given by the low-order 8 bits of the sum of pixel values in the image data area calculated by the adder 82. The purpose of the feature value is to identify each frame, and thus the feature value is not limited to the low-order 8 bits of the cumulative sum of the input signal (pixel values) over the image data area, but any other value correctly identifying each frame may be used as the feature value. For example, the sum of pixel values in a predetermined central area of the image data area may be used.

Next, referring to Fig. 9, an example of a configuration of the synchronization detector 42 is described in detail below.

Each of feature value buffers 101-1 to 101-3 is adapted to temporarily store a feature value input thereto. When a feature value is input to each of feature value buffers 101-1 to 101-3, a feature value currently stored therein is transferred to a feature value buffer (the feature value buffer 101-2 or 101-3) at the next position and also to the played-back frame feature value generator 102. Note that because there is no feature value buffer connected to the output of the feature value buffer 101-3, the feature value output from the feature value buffer 101-3 is supplied only to the played-back frame feature value generator 102. Each time the played-back frame feature value generator 102 receives the feature value of the current frame and the feature values of previous three frames from the feature value buffers 101-1 to 101-3, the played-back frame feature value generator 102 produces played-back frame information (played-back frame feature value) from the received feature values and outputs it to a comparator 103. That is, the played-back frame feature value generator 102 produces the played-back frame feature value identifying the played-back frame by using the feature values of four frames including the current frame and previous three frames.

A retrieved-frame feature value reading unit 104 sequentially reads four successive feature values at a time from the feature value/setting-information recording unit 62 and supplies them as retrieved-frame feature values to the comparator 103 and the detection unit 105. The comparator 103 compares the played-back frame feature values supplied from the played-back frame feature value generator 102 with the retrieved-frame feature values sequentially supplied from the retrieved-frame feature value reading unit 104. If the comparator 103 detects retrieved-frame feature values identical to the played-back frame feature values, the comparator 103 determines that synchronization is detected, and the comparator 103 notifies the detection unit 105 that synchronization is detected. If the detection unit 105 captures the retrieved-frame feature value supplied from the retrieved-frame feature value reading unit 104 at the same time as the time at which the synchronization notification is received from the comparator 103, and the detection unit 105 outputs this retrieved-frame feature value as a synchronization detection result.

Operations of the image processing apparatus 13 shown in Figs. 5 and 6 are described in further detail below with reference to flow charts shown in Figs. 10 to 14. The operation of the image processing apparatus 13 shown in Fig. 6 is basically similar to that of the image processing apparatus 13 shown in Fig. 5, and thus the operation is explained below, by way of example, for the image processing apparatus 13 shown in Fig. 5 with reference to flow charts shown in Figs. 10 to 14.

The image processing apparatus 13 is adapted to process an image which is input from, for example, a recording medium 11. More specifically, for example, the image recorded on the recording medium 11 is read by the image playback unit 12 (Fig. 5) and supplied to the image processing apparatus 13. In the image processing apparatus 13, the supplied image is temporarily stored on a frame-by-frame basis in the delay unit 48. Note that although it is assumed in the following explanation, by way of example, that the process is performed on a frame-by-frame basis, the process may be performed in different units. For example, the process may be performed on a field-by-field basis.

In step S11, the feature value extractor 41 performs a feature value extraction process to extract a feature value of one frame of image supplied from the image playback unit 12. More specifically, when one frame of image is supplied to the delay unit 48 for temporary storage, the same one frame of image is supplied to the feature value extractor 41. Thus, when a feature value of this image is extracted by the feature value extractor 41, the image is temporarily stored in the delay unit 48 during the process of feature value extraction.

The details of the feature value extraction process performed by the feature value extractor 41 in step S11 are described with reference to a flow chart shown in Fig. 11.

In step S31, the DFF 83 is reset at the frame start point S (Fig. 8).

In step S32, an unprocessed pixel is selected. In step S33, a determination is made as to whether the given pixel is out of the image data area. More specifically, pixels of one frame are sequentially read in a raster scan order, and the determination as to whether the pixel is out of the image data area is made each time an unprocessed pixel is input.

As can be seen from Fig. 8, a first pixel (at the frame start point S) is in the synchronization data area, and thus it is determined in step S33 that the pixel is out of the image data area. In this case, the process proceeds to step S35, in which the DFF 81 is reset. As a result, 0 is output as the pixel value from the DFF 81. In a case where a selected pixel is determined to be in the image data area, the process proceeds to step S34 in which the DFF 81 transfers the input pixel value to the adder 82 in synchronization with the clock signal.

In step S36, the adder 82 adds the input pixel value with the signal supplied from the DFF 83. The resultant value is supplied to the DFF 83. In step S37, low-order 8 bits of the value supplied as the addition result from the adder 82 to the DFF 83 are returned to the adder 82 from the DFF 83 in synchronization with the clock signal. In this case, although the low-order 8 bits of the addition result are also supplied to the output unit 84, the output unit 84 does not output the given data.

In step S38, a determination is made as to whether there are more unprocessed pixels. If so, the process returns to step S32, and the process from step S32 to S38 is performed repeatedly until all pixels of one frame have been processed. If it is determined in step S38 that the above-described process is completed for all pixels of one frame, then the processing flow proceeds to step S39. In step S39, the value supplied as the addition result from the DFF 83 to the output unit 84 is output from the output unit 84 as the feature value of the one frame of image. More specifically, low-order 8 bits of the sum of pixel values of pixels in the image data area of one frame are output from the output unit 84.

Thus, via the above-described process, pixel values are cumulatively added over the image data area of one frame of image signal, and low-order 8 bits of the resultant sum of pixel values are output as the feature value of this frame.

Referring again to the flow chart shown in Fig. 10, if the feature value extraction process performed by the feature value extractor 41 in step S11 is completed, then, in step S12, it is determined whether a recording flag is in an on-state. Note that when a process of recording operation information or setting-information (setting-value) is being performed, the recording flag is set in the on-state to indicate that the recording process is being performed.

Note that the determination as to whether the recording flag is in the on-state is made by the feature value/setting-information recording control unit 61. If it is determined in step S12 that the recording flag is in the on-state, the process proceeds to step S16. However, if it is determined in step S12 that the recording flag is in an off-state, the process proceeds to step S13.

In step S13, a process associated with synchronization is performed. The details of this process associated with synchronization are described below with reference to a flow chart shown in Fig. 12. The process associated with synchronization includes a synchronization detection process and a process of setting a setting-value depending on the result of the synchronization detection process. The synchronization detection process is performed in step S51, and the setting of the setting-value is performed in steps S52 to S56.

The synchronization detection process in step S51 is performed by the synchronization detector 42 (Fig. 9) to detect synchronization between a feature value of an image being currently played back and a feature value identifying a frame for which to perform a process specified by setting-information, as described in further detail below with reference to a flow chart shown in Fig. 13.

In step S71, the played-back frame feature value generator 102 of the synchronization detector 42 determines whether a feature value has been received from the feature value extractor 41. If no feature value has been received yet, step S71 is repeated until a feature value is received. If a feature value is extracted in the feature value extraction process in step S11 and the feature value of the frame of image being currently played back is supplied from the feature value extractor 41, then the process proceeds to step S72.

In step S72, the retrieved-frame feature value reading unit 104 initializes, to 0, a counter i for counting retrieved-frame feature values.

In step S73, the played-back frame feature value generator 102 acquires a newly supplied feature value and also acquires feature values stored in the feature value buffers 101-1 to 101-3. More specifically, when the new feature value is supplied to the feature value buffer 101-1, the previous feature value stored in the feature value buffer 101-1 is transferred to the played-back frame feature value generator 102 and the feature value buffer 101-2, and the feature value newly supplied to the feature value buffer 101-1 is stored therein.

Similarly, when the feature value is supplied from the feature value buffer 101-1 to the feature value buffer 101-2, the feature value currently stored in the feature value buffer 101-2 is transferred to the played-back frame feature value generator 102 and the feature value buffer 101-3, and the feature value supplied to the feature value buffer 101-2 is stored therein. When the feature value is supplied from the feature value buffer 101-2 to the feature value buffer 101-3, the feature value currently stored in the feature value buffer 101-3 is transferred to the played-back frame feature value generator 102, and the feature value supplied to the feature value buffer 101-3 is stored therein. As a result, the played-back frame feature value generator 102 receives the feature value of the newest frame and feature values of three previous frames stored in the feature value buffers 101-1 to 101-3. The played-back frame feature value generator 102 supplies, as the played-back frame feature value, the feature value of the frame being currently played back and the feature values of the three previous frames to the comparator 103.

In step S74, the retrieved-frame feature value reading unit 104 reads a feature value of an i-th frame as counted from the beginning and feature values of following three successive frames from the feature value/setting-information recording unit 62, and the retrieved-frame feature value reading unit 104 supplies these feature values as the retrieved-frame feature values to the comparator 103 and the detection unit 105.

In step S75, the comparator 103 compares the played-back frame feature values including the feature value of the frame being currently played back and the feature values of previous three frames supplied from the played-back frame feature value generator 102 with the retrieved-frame feature values including the feature value of the i-th frame as counted from the beginning and the feature values of following three frames read from the feature value/setting-information recording unit 62 and supplied from the retrieved-frame feature value reading unit 104.

In step S76, the comparator 103 determines whether the comparison result indicates that the feature values are identical. If it is determined in step S76 that no matching is found in feature values, the process proceeds to step S78. In step S78, the retrieved-frame feature value reading unit 104 determines whether the comparison of the played-back frame feature values has been made for all feature values stored in the feature value/setting-information recording unit 62. If it is determined in step S78 that the comparison is not completed for all feature values, the process proceeds to step S79. In step S79, the retrieved-frame feature value reading unit 104 increments the counter i by 1. The process then returns to step S74. That is, the process from step S74 to step S76 and steps S78 and S79 is performed repeatedly as long as no matching is found between the played-back frame feature values and the retrieved-frame feature values and the comparison is not completed for all feature values stored in the feature value/setting-information recording unit 62. In each iteration of this process, feature value data of four frames are shifted by one frame for the retrieved-frame feature values and the played-back frame feature values, and the comparison is made for these shifted data.

Note that in the present embodiment, the retrieved-frame feature values are sequentially given in the form of a set of feature values of four successive frames from the feature value extractor 41. In Fig. 15, each vertically-long rectangular block denotes a feature value of one frame. Four shaded blocks G1 are identical in feature value to four shaded blocks G2. Note that in the present embodiment, the played-back frame feature values are sequentially given in the form of a set of feature values of four successive frames, including the frame being currently played back, from the feature value extractor 41.

In the specific example shown in Fig. 15, when the set of played-back frame feature values G2 is compared with a set of retrieved-frame feature values G0, it is determined that there is no matching between them and that no synchronization is found. In this specific case, the comparison is not completed for all stored feature values, the counter i is incremented by 1, and the set of played-back frame feature values G2 is compared with a set of retrieved-frame feature values G0'. As described above, the comparison of the set of played-back frame feature values G2 with the set of retrieved-frame feature values is made repeatedly while shifting the set of retrieved-frame feature values by one frame at a time.

In the specific example, when the comparison is made between the set of played-back frame feature values G2 and the set of retrieved-frame feature values G1, the determination in step S76 is made that matching in feature values is found, and thus, in step S77, the comparator 103 notifies the detection unit 105 that matching in feature values is found. In response to the notification from the comparator 103, the detection unit 105 extracts a feature value at the first location in the current set of retrieved-frame feature values supplied from the retrieved-frame feature value reading unit 104, that is, an i-th feature value as counted from the beginning of whole stored feature values, and the detection unit 105 supplies the extracted feature value as synchronous position information to the feature value/setting-information reading unit 63.

In a case where it is determined in step S78 that the comparison is completed for all stored retrieved-frame feature values, the process proceeds to step S80. In step S80, the detection unit 105 outputs a signal indicating that there is no retrieved-frame feature value identical to the played-back frame feature value and thus no synchronization is found.

Via the above-described process, it is possible to synchronize the feature value stored in the feature value/setting-information recording unit 62 with the feature value of the image being played back. Note that in the above-described synchronization detection process, the synchronization detector 42 compares not only the feature value of one frame being currently played back with a feature value read from the feature value/setting-information recording unit 62, but feature values of four frames including the frame being currently played back are compared with feature values stored in the feature value/setting-information recording unit 62 thereby preventing an incorrect detection of synchronization due to sheer coincidence at a wrong position. By using feature values in the above-described manner, it becomes possible to detect synchronization in units of frames on the basis of feature values without using time codes of frames.

Referring again to the flow chart of the process associated with synchronization shown in Fig. 12, if the synchronization detection process in step S51 is completed, then in step S52, the feature value/setting-information reading unit 63 determines whether a feature value identical to the played-back frame feature value of a frame being currently played back has been found from the feature values stored in the feature value/setting-information recording unit 62. For example, in the case where the feature value/setting-information reading unit 63 receives the synchronous position information in step S77 of the flow chart shown in Fig. 13, the feature value/setting-information reading unit 63 determines that a feature value identical to the played-back frame feature value of the frame being currently played back has been found from the feature values stored in the feature value/setting-information recording unit 62, and the process proceeds to step S53.

In step S53, the feature value/setting-information reading unit 63 determines whether the feature value/setting-information recording unit 62 includes setting-information corresponding to the detected feature value in synchronization. Because feature values corresponding to no setting-information can be stored in the feature value/setting-information recording unit 62, it is necessary to make the determination in step S53 to check whether there is setting-information corresponding to the feature value.

In a case where it is determined in step S53 that there is no setting-information corresponding to the feature value, the process proceeds to step S54 in which a normal value is set as the setting-value. Herein, the "normal" value refers to a default value employed when no operation is performed to designate a process to be applied to the image.

In the case where the normal value is set as the setting-value in step S54, the process proceeds to step S55 in which the setting-value is applied to the image. When the normal value is set as the setting-value (setting-information) as in the present case, the feature value/setting-information reading unit 63 does not supply setting-information to the selector 47. In this case, the selector 47 does not receive the setting-information from either the setting-information determination unit 46 or the feature value/setting-information reading unit 63, and thus setting-information specifying a process to be performed is not supplied to the applying unit 49. As a result, the applying unit 49 directly displays one frame of image temporarily stored in the delay unit 48 on the display unit 15.

On the other hand, in the case where the determination made in step S53 is that there is stored setting-information corresponding to the feature value, the process proceeds to step S56 in which the setting-information (the setting-value) is read.

When the setting-value is read, the setting-value is applied to the image in step S55. That is, in this case, the feature value/setting-information reading unit 63 reads the setting-information corresponding to the feature value of the synchronous position information from the feature vafue/setting-information recording unit 62 and supplies it to the selector 47. In the present case, the selector 47 receives the setting-information from the feature value/setting-information reading unit 63 but no setting-information from the setting-information determination unit 46, and thus the selector 47 transfers the setting-information received from the feature value/setting-information reading unit 63 to the applying unit 49. According to this setting-information, one frame of image stored in the delay unit 48 is processed, and a resultant image is displayed on the display unit 15.

As described above, in the state in which synchronization is performed, the image is processed according to the setting-value, and the resultant image is displayed.

Referring again to the flow chart shown in Fig. 10, if the above-described process associated with synchronization in step S13 is completed, then, in step S14, the operation information analysis unit 45 determines whether an image-processing command has been issued by a user by operating the remote control 14 (hereinafter, such an operation performed by a user to designate an effect to be applied to an image will be referred to simply as an "effect/edit control operation"). For example, when an image is being displayed on the display unit 15, if the user issues a 2X-zoom command, an optical signal corresponding to the 2X-zoom command is emitted from the remote control 14. If the light receiver 44 receives the optical signal, the light receiver 44 outputs an electrical signal corresponding to the received optical signal to the operation information analysis unit 45. Accordingly, the operation information analysis unit 45 determines that an effect/edit control operation has been performed, and thus the process proceeds to step S15.

In step S15, the recording flag is set. The process comes in step S15 only in a situation in which the determination made in step S12 is that the recording flag is in the off-state and the determination made in step S14 is that an image-processing command has been issued by a user. Therefore, in this case, it is necessary to start recording operation information or setting-information. Thus, in step S15, the recording flag is set.

After the recording flag is set in step S15, the process proceeds to step S16 to perform the effect/edit process in accordance with the effect/edit control operation. The process also comes in step S16 when it is determined in step S12 that the recording flag is in the on-state. That is, step S16 is performed when the recording flag is in the on-state. In other words, when it is determined in step S12 that the recording flag is in the off-state and it is determined in step S14 that no effect/edit control operation has been performed by a user, it is not necessary to record the operation information or setting-information, and thus step S16 is not performed.

The effect/edit process performed in accordance with the effect/edit control operation in step S16 is described in detail with reference to a flow chart shown in Fig. 14. In step S91, the operation information analysis unit 45 analyzes the signal supplied from the light receiver 44 to detect operation information and supplies the detected operation information to the setting-information determination unit 46.

In step S92, it is determined whether recording is ended. Note that there is a possibility that the operation information acquired in step S91 indicates a record end command. More specifically, for example, if the effect/edit control operation by the user is ended, and an end button is finally pressed by the user, then a signal generated in response to pressing the end button is received by the light receiver 44, and thus it is determined that the received operation information indicates that recording should be ended.

In the case where it is determined in step S92 that recording is not ended, the process proceeds to step S93. That is, if it is determined that recording is still being performed (because the effect/edit control operation is still being performed), the process proceeds to step S93 to produce the setting-value in accordance with the content of the operation information.

More specifically, in step S93, in accordance with the operation information, the setting-information determination unit 46 determines setting-information indicating a process to be performed on the image by the applying unit 49, and the setting-information determination unit 46 supplies the setting-information to the feature value/setting-information recording control unit 61 and the selector 47. For example, if a 2X-zoom command is issued, the setting-information is determined so as to instruct the applying unit 49 to perform a 2X-zoom process on the image, and the resultant setting-information is supplied to the feature value/setting-information recording control unit 61 and the selector 47.

In step S94, the feature value/setting-information recording control unit 61 records the feature value supplied from the feature value extractor 41 in the feature value/setting-information recording unit 62 and also records the setting-information supplied from the setting-information determination unit 46 in association with the feature value. Via the above-described process, feature values and setting-information are stored in the feature value/setting-information recording unit 62. More specifically, for example, as shown in Fig. 16, when a feature value C1 is recorded as an address A, setting-information E1 corresponding to the feature value C1 is recorded at an address equal to the address A times m, that is, m x A. Note that Fig. 16 shows locations of addresses in the feature value/setting-information recording unit 62 where feature values and setting-information are recorded.

Similarly, for a feature value C2 and corresponding setting-information E2, the feature value/setting-information recording control unit 61 records the feature value C2 at an address B and the setting-information E2 at an address m x B. For a feature value C3 and corresponding setting-information E3, the feature value/setting-information recording control unit 61 records the feature value C3 at an address C and the setting-information E2 at an address m x C. By recording setting-information in association with corresponding feature values as described above, it is possible to identify an address of setting-information corresponding to a feature value from the address of the feature value. This makes it possible to easily read setting-information corresponding to each feature value stored in the setting-information recording unit 62. Note that feature values and corresponding setting-information are continuously recorded over a period during which a process to be performed on the image is continuously specified. Note that each of the feature values C1 to C3 shown in Fig. 16 is not limited to that corresponding to one frame, but as many feature values may be continuously recorded as there are frames in a period during which a process is continuously specified. As for setting-information, each of the setting-information E1 to E3 shown in Fig. 16 is not limited to that corresponding to a feature value of one frame, but as many pieces of setting-information may be recorded at each address corresponding to the address of each feature value C1 to C3 as there are frames in the period during which the process is continuously specified.

In step S95, the selector 47 transfers the setting-information received from the setting-information determination unit 46 to the applying unit 49. The applying unit 49 processes the image stored in the delay unit 48 in accordance with the setting-information supplied from the selector 47. Thus, the resultant image produced by processing the original image in accordance with the command issued by the user is displayed on the display unit 15.

On the other hand, in the case where it is determined in step S92 that recording is ended, the process proceeds to step S96. In step S96, the setting-value is set to a normal value. In step S97, processing is performed in accordance with the setting-value. In this specific case, because the normal value is employed as the setting-value, the feature value/setting-information reading unit 63 does not supply setting-information to the selector 47. Therefore, the selector 47 does not receive the setting-information from either the setting-information determination unit 46 or the feature value/setting-information reading unit 63, and thus setting-information specifying a process to be performed is not supplied to the applying unit 49. As a result, the applying unit 49 directly displays one frame of image temporarily stored in the delay unit 48 on the display unit 15.

In step S98, the recording flag is reset. In response, the recording of setting-information is ended.

As described above, as long as a command specifying a process to be performed on an image is continuously output by operating the remote control 14 by a user, the process from step S91 to S95 is performed repeatedly, whereby setting-information indicating the process to be performed is stored in association with feature values in the feature value/setting-information recording unit 62. Note that this iterative process from step S91 to step S95 is performed in the recording mode described above with reference to Figs. 1 and 2.

Referring back to the flow chart shown in Fig. 10, if the effect/edit process in step S16 is ended, the image processing is ended. As described above, the process of the flow chart shown in Fig. 10 is performed for each frame. That is, each time a new frame is input, the process of the flow chart shown in Fig. 10 is performed for the input frame. Thus, as described above, as long as a command specifying a process to be performed on an image is continuously output by operating the remote control 14 by a user, setting-information indicating the process to be performed is stored in association with feature values in the feature value/setting-information recording unit 62.

By using the image processing apparatus 13 shown in Fig. 5 or 6, it is possible to edit an image recorded on the recording medium 11 a plurality of times by performing the above-described process the plurality of times. By using the image processing apparatus 13 shown in Fig. 5 or 6, it is possible to play back the image in the form edited the plurality of times from the recording medium 11.

As described above with reference to Fig. 1 or 2, there is a possibility that an abrupt change in the setting-value occurs at the end of the recording depending on how editing is performed. In such a case, for example, an abrupt change in zoom ratio or the like can occur during playback of an image. Such an abrupt change during playback gives a discomfort impression to a user. The above problem can be avoided by performing recording without producing an abrupt change in setting-values by using the image processing apparatus 13 as described below. Hereinafter, a function of controlling the setting-value to have no abrupt change will be referred to as a smooth end function.

Referring to Figs. 17A to 17C, the smooth end function is described below. Fig. 17A shows a first-time effect/edit control operation and a setting-value recorded in accordance with the effect/edit control operation. Fig. 17B shows a second-time effect/edit control operation and a setting-value recorded in accordance with the effect/edit control operation. Fig. 17C shows setting-values recorded as a total result of the above process. In each of Figs. 17A and 17B, the setting-value is shown in the upper part of the figure, and the effect/edit control operation is shown in the lower part of the figure. Figs. 17A to 17C are drawn in a similar manner to Figs. 2A to 2C so that it is possible to easily understand a difference between the recording method by the smooth end function and the conventional recording method.

In the first-time effect/edit control operation, if an operation signal such as that shown in the lower part of Fig. 17A is output from the remote control 14, a setting-value such as that plotted in the upper part of Fig. 17A is recorded. Note that the setting-value is given by the integral of an operation value of the remote control 14. In Figs. 17A and 17B, a "positive operation" refers to a positive effect/edit control operation such as enlarging of an image, and a "negative operation" refers to a negative effect/edit control operation such as reducing of an image.

In the example shown in Fig. 17A, a positive operation is performed over a period from an operation start time T1 to a time T11 as shown in the lower part of Fig. 17A, and the integral of the operation value is recorded as the setting-value, which gradually increases with time as shown in the lower part of Fig. 17B. In a period from the time T11 to a time T12, a negative operation is performed, and accordingly the recorded setting-value gradually decreases. After the time T12, positive and negative operations are performed, and the setting-value is recorded in accordance with the operations.

If an operation end button (a record end button) of the remote control 14 is pressed at a time T14, the recording of the setting-value is ended. Note that when the recording is ended at the time T14, the setting-value does not return to the normal value immediately, but the setting-value is reduced gradually to the normal value in a period from T14 to T2 by the smooth end function, and the resultant setting-value is recorded. As a result of the above-described process including the smooth end process, the setting-value is recorded as shown in the upper part of Fig. 17A.

Note that the normal value refers to a default setting-value, and setting-information corresponding to the normal value is referred to as normal setting-information.

In the specific example shown in Fig. 17B, the second-time effect/edit control operation is performed on the same content as that subjected to the first-time effect/edit control operation such that a positive operation is performed over a period from a time T3 to a time T21, and a negative operation is performed over a period from the time T21 to a time T4. As a result, a setting-value is recorded in the period from the time T3 to the time T4 as shown in the upper part of Fig. 17B.

In the second-time effect/edit control operation, the setting-value given by the integral of the operation value is added to the already-recorded setting-value obtained at the point of time T3 in the first-time effect/edit control operation. When the operation is ended at the time T4, the setting-value is gradually reduced in a period from the time T4 to a time T22 to a value (A) obtained at the time T4 in the first-time effect/edit control operation.

As a result of a modification of the setting-value according to the second-time effect/edit control operation, the setting-value is recorded as represented by a solid line shown in the upper part of Fig. 17B (that is, the setting-value is modified to a value represented by the solid line). Thus, after the completion of the second-time effect/edit control operation, the setting-value is recorded as shown in Fig. 17C as a total result of the process.

In the recorded result shown in Fig. 17C, unlike the recorded result shown in Fig. 2C in which abrupt changes in setting-value occur at times T4 and T2, the setting-value gradually changes in the period from the time T4 to the time T22 and the period from the time T14 to the time T2.

As described above, by the smooth end function, the setting-value is gradually changed at the end of the recording operation such that an abrupt change in setting-value does not occur.

Fig. 18 shows an example of a configuration of an image processing apparatus obtained by modifying the configuration of the image processing apparatus 13 shown in Fig. 5 so as to have the smooth end function, and Fig. 19 shows an example of a configuration of an image processing apparatus obtained by modifying the configuration of the image processing apparatus 13 shown in Fig. 6 so as to have the smooth end function.

In Fig. 18, similar parts to those in Fig. 5 are denoted by similar reference numerals, and a duplicated explanation thereof is omitted herein. The image processing apparatus 13 shown in Fig. 18 is similar to the image processing apparatus 13 shown in Fig. 5 except that it additionally includes a smooth end controller 201 adapted to execute a smooth end process.

The smooth end controller 201 receives/transmits setting-information from/to the setting-information determination unit 46. Information output from the smooth end controller 201 is supplied to the feature value/setting-information recording control unit 61 and the selector 47.

The setting-information determination unit 46 converts the input operation into setting-information and supplies the resultant setting-information to the smooth end controller 201. The smooth end controller 201 directly outputs the setting-information received from the setting-information determination unit 46 except when the end command is issued by the input operation. In the case where the end command is issued by the input operation, the smooth end controller 201 produces setting-information by interpolating between the last one of the input operation values stored in the feature value/setting-information recording unit 62 and the output from the feature value/setting-information recording unit 62. The produced setting-information is supplied to the applying unit 49 via the selector 47, and the applying unit 49 processes the image in accordance with the supplied setting-information. The setting-information is also supplied to the feature value/setting-information recording control unit 61 and recorded in the feature value/setting-information recording unit 62.

When a transition occurs in the setting-information output from the feature value/setting-information recording unit 62 as in the case where a tansition occurs when the setting-information is returned from the end value given in the second-time effect/edit control operation to the original value obtained in the first-time effect/edit control operation as described above with reference to Figs. 17A to 17C, the smooth end controller 201 smoothly connects the two pieces of setting-information.

Now, referring to Fig. 19, an example of a modification of the image processing apparatus 13 shown in Fig. 6 so as to have the smooth end function is described below. In Fig. 19, similar parts to those in Fig. 6 are denoted by similar reference numerals, and a duplicated explanation thereof is omitted herein. The image processing apparatus 13 shown in Fig. 19 is similar to the image processing apparatus 13 shown in Fig. 6 except that it additionally includes a smooth end controller 211 adapted to execute a smooth end process. The smooth end controller 211 is adapted to receive/transmit setting-information from/to the setting-information determination unit 153.

At a transition where original setting-information produced by the setting-information determination unit 153 changes discontinuously, the smooth end controller 211 modifies the setting-information by interpolation such that it gradually change.

Operations of the image processing apparatus 13 shown in Figs. 18 and 19 are described in further detail below with reference to flow charts shown in Figs. 20 to 23. The operation of the image processing apparatus 13 shown in Fig. 19 is basically similar to that of the image processing apparatus 13 shown in Fig. 18, and thus the operation is explained below, by way of example, for the image processing apparatus 13 shown in Fig. 18 with reference to flow charts shown in Figs. 20 to 23.

First, referring to Fig. 20, image processing performed by the image processing apparatus 13 shown in Fig. 18 is described. Basically, the image process is performed in a similar manner to that performed by the image processing apparatus 13 shown in Fig. 5, described above with reference to the flow chart shown in Fig. 10. More specifically, steps S201 to S206 are similar to steps S11 to S16 described in detail above with reference to the flow chart shown in Fig. 10, and thus the following discussion will be focused on differences.

In step S201, the feature value extractor 41 performs a feature value extraction process in a similar manner to that described above in the flow chart shown in Fig. 11.

In step S202, it is determined whether the recording flag is in the on-state. If it is determined that the recording flag is in an off-state, the process proceeds to step S203. In step S203, a process associated with synchronization is performed. The details of this process associated with synchronization in step S203 are described below with reference to a flow chart shown in Fig. 21.

The flow chart shown in Fig. 21 is basically similar to that shown in Fig. 12 except that there is an additional step S235. After the setting-value is set to the normal value in step S234 or after setting-information (setting-value) is read from the feature value/setting-information recording unit 62 in step S237, the process proceeds to step S235 to perform a smooth end process. In step S236, the image is processed according to the setting-values subjected to the smooth end process.

The smooth end process performed in step S235 is described below with reference to a flow chart shown in Fig. 22. Note that the process shown in the flow chart of Fig. 22 is performed by the smooth end controller 201. In step S251, it is determined whether a smooth end flag is in an on-state. Note that the smooth end flag is set when the smooth end function should be executed, that is, when an edit end command is issued by a user.

In a case where it is determined in step S251 that the smooth end flag is in the off-state, it is not necessary to perform the smooth end process, and thus the smooth end process is ended.

On the other hand, in a case where it is determined in step S251 that the smooth end flag is in the on-state, the process proceeds to step S252. In step S252, the setting-value is set to a value between the setting-value for a previous frame and a current setting-value. Note that the setting-value for the previous frame refers to the setting-value for the frame immediately previous to a current frame for which the process shown in Fig. 20 is being performed, and the feature value of the previous frame has been stored in the feature value/setting-information recording unit 62. The current frame refers to a frame being currently processed, and the setting-value for the current frame has been set in step S234 or step S237 (Fig. 21).

In step S252, the setting-value is set to a value between the setting-value for the previous frame and the setting-value for the current frame. For example, when the setting-value for the previous frame is greater than the setting-value for the current frame, the setting-value is set to a value equal to the setting-value for the previous frame minus a predetermined value. When the setting-value for the previous frame is smaller than the setting-value for the current frame, the setting-value is set to a value equal to the setting-value for the previous frame plus a predetermined value. As described above, the setting-value for the current frame is set to a value equal to the setting-value for the previous frame plus or minus the predetermined value.

In step S253, it is determined whether the setting-value for the previous frame is equal to the setting-value for the current frame. Instead of determining whether the setting-value for the previous frame is equal to the setting-value for the current frame, the determination may be made as to whether the difference between the setting-value for the previous frame and the setting-value for the current frame is within a predetermined range.

In a case where it is determined in step S253 that the setting-value for the previous frame is not equal to the setting-value for the current frame, step S254 is skipped and the smooth end process for the frame of interest is ended. On the other hand, in a case where it is determined in step S253 that the setting-value for the previous frame is equal to the setting-value for the current frame, the process proceeds to step S254, in which the smooth end flag is reset.

In a period in which the smooth end flag is in the on-state, the process in step S252 is performed for each frame, and the setting-value approaches the setting-value recorded in the process performed in response to the previous effect/edit control operation. Thus, it is possible to prevent an abrupt change in the setting-value as described in further detail below with reference to Fig. 24.

If the effect/edit control operation is ended at a frame F1, and if the setting-value for the frame F1 is set to R1, then, in the conventional technique, the setting-value for a next frame F2 is returned to a value R12 already recorded for the frame F2. However, in the present embodiment, unlike the conventional technique, the smooth end process (step S252) is performed to set the setting-value for the frame F2 to a value R2 between the setting-value R1 and the setting-value R12, and more specifically to a value equal to the setting-value R1 minus the predetermined value.

For a frame F3, the setting-value is set to a value R3 equal to the setting-value R2 minus the predetermined value. The setting-value for a frame F4 is set to a value R4, and the setting-value for a frame F5 is set to a value R5 in a similar manner.

In the present example, the difference between the setting-value R5 for the frame F5 and the original recorded setting-value R15 is within the predetermined range, and thus the setting-value for a next frame F6 is set to value equal to a setting-value R16 originally set and recorded for the frame F6. In the present embodiment, as described above, the setting-value is changed gradually to the original value set in the process performed in response to the previous effect/edit control operation, without abruptly changing the setting-value to the original value.

Referring back to the flow chart shown in Fig. 21, if the smooth end process in step S235 is completed, the process proceeds to step S236 to apply the setting-value to the image. If the process associated with synchronization is completed, the process proceeds to step S204 (Fig. 20).

In step S204, it is determined whether an effect/edit control operation (for editing) is performed by a user. If it is determined that an effect/edit control operation is performed, the process proceeds to step S205. In step S205, the recording flag is set. In step S206, an effect/edit process is performed in accordance with the effect/edit control operation. The details of the process performed in step S206 in accordance with the effect/edit control operation are described below with reference to a flow chart shown in Fig. 23.

Steps S271 to S278 in Fig. 23 are similar to steps S91 to S98 in Fig. 14, and thus a duplicated explanation thereof is omitted herein. In step S279, the smooth end flag is set. The process comes in step S279 only when the determination made in step S272 is that recording has been ended. The recording is ended when an operation end button or the like is pressed by a user.

In the conventional technique, when an operation (edit) end command is issued by a user, the setting-value returns to an original already-recorded value, as described above. However, such an abrupt change in setting-value can cause an undesirable abrupt change in image. In the present embodiment, to avoid the above problem, the smooth end process is performed. The smooth end process is performed when the smooth end flag is in the on-state. Thus, when the operation end command is issued by a user, the smooth end flag is set in step S279.

If the smooth end flag is set, then an affirmative answer is given to the determination in step S251 (Fig. 22) as to whether the smooth end flag is in the on-state. As a result, steps S252 and S253 are performed. Thus, the setting-value is changed gradually to the original already-recorded value.

As described above, execution of the smooth end process prevents an abrupt change in setting-value when editing is performed a plurality of times. Thus, an abrupt change in image is prevented.

An automatic effect/edit control operation end function is described below. The automatic effect/edit control operation end function makes it possible to automatic end an effect/edit control operation without a user having to perform a particular operation.

Referring to Figs. 25A to 25C, the automatic effect/edit control operation end function is described. Fig. 25A shows a first-time effect/edit control operation and a setting-value recorded in accordance with the effect/edit control operation. Fig. 25B shows a second-time effect/edit control operation and a setting-value recorded in accordance with the effect/edit control operation. Fig. 25C shows a setting-value recorded as a total result of the above process. In each of Figs. 25A and 25B, the setting-value is plotted in the upper part of the figure, and the operation result is shown in the lower part of the figure.

In the specific example shown in Fig. 25A, a first-time effect/edit control operation is performed on a content in a period from a time T1 to a time T2, and a setting-value is recorded in accordance with the effect/edit control operation in the period from T1 to T2. More particularly, a positive operation is performed in a period from T1 to T11, a negative operation is performed in a period from T11 to T12, a positive operation is performed in a period from T12 to T13, and a negative operation is performed in a period from T13 to T14. As a result, a setting-value is recorded as shown in the upper part of Fig. 25A.

In the present example, no effect/edit control operation is performed in a period from T14 to T2, that is, the period from T14 to T2 is a no-operation period. If the length of the no-operation period is greater than a predetermined value, the setting-value returns to a normal value without needing a command issued by a user.

After the setting-value was recorded according to the first-time effect/edit control operation, if a second-time effect/edit control operation is performed as shown in Fig. 25B, then a setting-value is recorded as shown in Fig. 25C. That is, in the example shown in Fig. 2B, the second-time effect/edit control operation is performed such that a positive effect/edit control operation is performed in a period from a point of time T3 to a point of time T21, a negative effect/edit control operation is performed in a period from the point of time T21 to a point of time T4, and no effect/edit control operation is performed after that. If no operation is performed over a period with a length equal to or greater than a predetermined value, that is, if the length of the period from T4 to T22 is equal to or greater than the predetermined value, the setting-value returns to a normal value without needing a command issued by a user.

When an operation end command is not issued by a user, if no effect/edit control operation is performed over a predetermined period, the setting-value is gradually returned to the original value.

As described above, the automatic effect/edit control operation end function makes it possible to determine whether the effect/edit control operation has been ended, and to automatically end the effect/edit control operation even if an effect/edit control operation end command is not issued by a user in such a manner that the setting-value is gradually changed to the original value without changing it abruptly.

Fig. 26 shows an example of a configuration of an image processing apparatus obtained by modifying the configuration of the image processing apparatus 13 shown in Fig. 5 so as to have the automatic operation end function, and Fig. 27 shows an example of a configuration of an image processing apparatus obtained by modifying the configuration of the image processing apparatus 13 shown in Fig. 6 so as to have the automatic operation end function.

In Fig. 26, similar parts to those in Fig. 5 are denoted by similar reference numerals, and a duplicated explanation thereof is omitted herein. The image processing apparatus 13 shown in Fig. 26 is similar to the image processing apparatus 13 shown in Fig. 5 except that it additionally includes an automatic operation end controller 301 adapted to execute the automatic operation end process.

The automatic operation end controller 301 is adapted to transmit/receive data to/from the setting-information determination unit 46, the storage block 43, and the selector 47.

In Fig. 27, similar parts to those in Fig. 6 are denoted by similar reference numerals, and a duplicated explanation thereof is omitted herein. The image processing apparatus 13 shown in Fig. 27 is similar to the image processing apparatus 13 shown in Fig. 6 except that it additionally includes an automatic operation end controller 311 adapted to execute the automatic operation end process.

The automatic operation end controller 311 is adapted to transmit/receive data to/from the storage block 151 and the selector 152.

Image processing performed by the image processing apparatus 13 shown in Fig. 26 is described below with reference to flow charts shown in Figs. 28 to 30.

In step S301, a feature value extraction process is performed. The feature value extraction process in step S301 is performed in a similar manner as described above with reference to the flow chart shown in Fig. 11, and thus a duplicated explanation thereof is omitted herein.

In step S302, the process associated with synchronization is performed. The details of this process associated with synchronization in step S302 are described below with reference to a flow chart shown in Fig. 29. Steps S321 to S325 are similar to steps S51 to S54 and S56 described above with reference to the flow chart shown in Fig. 12, and thus a duplicated explanation thereof is omitted herein. Note that the process associated with synchronization does not include applying the setting to the image.

After the synchronization detection process, the process proceeds to step S303 in the flow chart shown in Fig. 28. In step S303, it is determined whether the recording flag is in the on-state. In the smooth end process described above with reference to the flow chart shown in Fig. 20, it is determined in step S202 whether the recording flag is in the on-state, and the process associated with synchronization is performed in step S203.

The process of the flow chart shown in Fig. 28 is different from the process of the flow chart shown in Fig. 20 in that the step of determining whether the recording flag is in the on-state, that is, whether the setting-value is being recorded and the step of performing the process associated with synchronization are exchanged with each other in terms of processing order.

By performing the process associated with synchronization before the determination is made as to whether the recording flag is in the on-state, that is, by performing the process associated with synchronization regardless of whether the recording flag is in the on-state, it becomes possible to acquire a setting-value in a state in which playback is performed without performing recording.

In a case where it is determined in step S303 that the recording flag is in the off-state, the process proceeds to step S304 to apply the read setting-value. Because the process associated with synchronization has been performed in step S302 regardless of whether recording flag is in the on-state, and thus the setting-value has been set in step S324 or S325 (Fig. 29). Thus, in the present step S304, the setting-value is applied to the image.

In step S305, it is determined whether an effect/edit control operation is performed. If it is determined that an effect/edit control operation has been performed, the process proceeds to step S306 in which the recording flag is set. The process then proceeds to step S307 to perform the effect/edit process. The details of the process associated with synchronization in step S307 are described below with reference to a flow chart shown in Fig. 30.

In step S341, the operation information analysis unit 45 analyzes the signal supplied from the light receiver 44 to identify operation information and supplies the identified operation information to the setting-information determination unit 46. In step S342, it is determined whether the operation information acquired in step S341 is of a record end command.

In a case where it is determined in step S342 that the acquired operation information is not of the record end command, the process proceeds to step S343. However, if it is determined in step S342 that the acquired operation information is of the record end command, the process proceeds to step S347.

In step S343, it is determined that an automatic end condition is satisfied. Note that the automatic end condition is satisfied, as described above with reference to Figs. 25A to 25C, when no effect/edit control operation is performed over a period with a length equal to or greater than the predetermined value (that is, when the length of the no-operation period is equal to or greater than the predetermined value). More specifically, the automatic operation end controller 301 (Fig. 26) measures the time during which the setting-value supplied from the setting-information determination unit 46 remains unchanged, and determines whether the measured time has reached the predetermined value (that is, whether the setting-information has remained unchanged over a period with a predetermined length or longer), thereby determining whether the automatic end condition is satisfied.

Alternatively, the automatic end condition may be