The present invention relates to playback apparatuses, playback methods, programs, recording media, data structures, and manufacturing methods for recording media. More particularly, the invention relates to a playback apparatus, a playback method, a program, a recording medium, a data structure, and a manufacturing method for the recording medium, which are suitably used when, not only content of a main AV stream, but also various types of content items, such as audio and video content items, of streams or data files different from the main AV stream can be played back.

In DVD (Digital Versatile Disc) video standards, interactive operations can be performed, that is, users can switch sound or subtitles when playing back AV (Audio Visual) content, such as a movie, recorded in a recording medium (for example, see Non-Patent Document 1). More specifically, the user operates a sound switch button 11 or a subtitle switch button 12 of a remote controller 2 to switch sound or subtitles of AV content displayed on a display device 1 shown in Fig. 1. For example, if the user operates the sound switch button 11 when sound 1 is set in the initial state in the display device 1, sound 1 is switched to sound 2, as shown in Fig. 2.

AV content based on the DVD video is recorded in the form of a MPEG (Moving Picture Experts Group) 2 program stream. In the MPEG2 program stream, as shown in Fig. 3, a video stream (indicated by "video" in Fig. 3), a plurality of audio streams (indicated by "audio 1", "audio 2", and "audio 3" in Fig. 3), and a plurality of sub-picture streams (indicated by "sub-picture 1, "sub-picture 2", and "sub-picture 3") are multiplexed such that the audio streams and the sub-picture streams are AV-synchronized with the video stream. The sub-picture streams (sub-pictures 1, 2, and 3) are streams in which bitmap images are run-length coded, and are mainly used for subtitles.

Generally, a plurality of audio streams are used for recording sound of different languages, and a plurality of sub-picture streams are used for recording subtitles of different languages. The user can interactively select sound or subtitles of a desired language by using the remote controller 2 while video is being played back.

The DVD video defines a table structure, provided to users, indicating the relationship between sound numbers and subtitle numbers for a plurality of audio streams (audio 1, 2, and 3) and a plurality of sub-picture streams (sub-pictures 1, 2, and 3) in a program stream.

Fig. 4 illustrates an example of such a stream number table indicating the relationship between audio signals and subtitle signals provided to the user. In this table, the sound numbers are referred to as "A_SN (Audio Stream Numbers)", and the subtitle numbers are referred to as "S_SN (Sub-picture Stream Numbers)". In the stream number table shown in Fig. 4, each of the audio streams is provided with A_SN, and each of the sub-picture streams is provided with S_SN. A_SN=1: audio 2, A_SN=2: audio 1, and A_SN-3: audio 3. Also, S_SN=1: sub-picture 3, S_SN=2: sub-picture 1, and S_SN=3: sub-picture 2. In this case, a smaller number of A_SN or S_SN indicates an audio signal or a subtitle signal to be provided to users with higher priority. That is, A_SN=1 is an audio stream played back as a default, and S_SN=1 is a sub-picture stream played back as a default.

More specifically, sound 1 played back in the initial state in Fig. 1 is audio 2, which is A_SN=1 (Fig. 4), and sound 2 played back after being switched from sound 1 in Fig. 2 is audio 1, which is A_SN=2.

[Non-Patent Document 1]

DVD Specifications for Read-Only Disc Part 3; Version 1.1

A technique for using Sub Paths to play back sound which is additionally recorded (so-called "dubbed sound") is known (for example, see Patent Document 1).

[Patent Document 1]

Japanese Unexamined Patent Application Publication No. 2002-158972 .

According to the DVD video, however, when the user switches sound or subtitles while playing back a video program stream, the user can select only from audio streams or sub-picture streams multiplexed into the program stream which is currently played back. That is, when playing back the MPEG2 program stream shown in Fig. 3, the user can select only from audio 1 through audio 3 when switching sound.

Accordingly, even if another stream having audio streams and subtitles different from a program stream which is currently played back is available, the user cannot switch sound or subtitles to the audio streams or subtitles in the different stream. Accordingly, the extensibility in selecting streams is low.

It can also be considered that in the near future, content to be switched will not be restricted to sound, but will be extended to various types of content items, such as video. Since it is difficult to select content items from a stream different from a stream which is being played back when switching such various types of content, the extensibility in selecting streams will become low.

Although Patent Document 1 discloses a technique for using Sub Paths for playing back sound which is additionally recorded (for example, so-called "dubbed sound"), it does not disclose a configuration in which two streams of the same type, i.e., two audio streams or two video streams, are combined and played back is not disclosed.

The present invention has been made in view of such a background. It is an object of the present invention to select various types of content, such as sound and video, from streams or data files different from a main AV stream so that the selected content can be played back together with the main AV stream.

A playback apparatus according to one aspect of the present invention is a playback apparatus for playing back streams. The playback apparatus includes: obtaining means for obtaining playback management information including a main playback path indicating a position of at least one stream on a time axis, at least one sub playback path indicating a position of at least one stream on the time axis, and selection information for selecting at least one category of stream, which is to be played back, from among at least one stream referred to by the main playback path and for selecting at least one category of stream, which is to be played back, referred to by one of at least one sub playback path; receiving means for receiving a selection of streams to be played back based on the selection information included in the playback management information obtained by the obtaining means; reading means for reading, if the selection of streams to be played back is received by the receiving means, by referring to the main playback path, at least one category of stream selected from among at least one stream referred to by the main playback path and received by the receiving means, and for reading, by referring to the corresponding sub playback path, at least one category of stream selected from among at least one stream referred to by at least one sub playback path and received by the receiving means; and combining means for combining at least two streams of a predetermined category, if at least two streams are included in at least one category of stream referred to by the main playback path and read by the reading means and in at least one category of stream referred to by the corresponding sub playback path and read by the reading means. The selection information includes information indicating a combination of a first stream of the predetermined category referred to by the main playback path and a second stream of the predetermined category referred to by the corresponding sub playback path, the receiving means receives the selection of streams to be played back based on the selection information, the reading means reads, if the selection of the first stream and the second stream to be played back is received by the receiving means, the first stream of the predetermined category referred to by the main playback path and the second stream of the predetermined category referred to by the corresponding sub playback path, and the combining means combines the first stream and the second stream of the predetermined category read by the reading means.

The playback apparatus may further include decoding means for decoding at least one category of stream referred to by the main playback path and read by the reading means and at least one category of stream referred to by the corresponding sub playback path and read by the reading means.

The streams of the predetermined category to be combined by the combining means may be audio streams or video streams.

The stream of the predetermined category referred to by the main playback path may be a first video stream, and the selection information may further include information defining a combination of a first audio stream associated with the first video stream and a second audio stream different from the first audio stream.

The receiving means may receive the selection of a stream to be played back from among: the first stream referred to by the main playback path; the second stream referred to by the corresponding sub playback path; a combination of the first stream referred to by the main playback path and the second stream referred to by the corresponding sub playback path, the second stream being included in a file different from a file including the first stream; a combination of the second streams referred to by the corresponding sub playback paths; and a combination of the first stream referred to by the main playback path and the second stream referred to by the corresponding sub playback path, the second stream being included in an identical file including the first stream.

The playback management information obtained by the obtaining means may further include information concerning a combination of two streams of a predetermined category referred to by the corresponding sub playback paths, the reading means may read, if the selection of the combination of the two streams of the predetermined category to be played back is received by the receiving means, the two streams of the predetermined category referred to by the corresponding sub playback paths, and the combining means may combine the two streams of the predetermined category read by the reading means.

The selection information may further include information indicating whether streams to be combined are entered, the receiving means may receive, based on the information indicating whether streams to be combined are entered, a selection of a combination of streams of an identical category referred to by at least one of the main playback path and the corresponding sub playback path or receives a selection of a single stream of an identical category, and the reading means may read the entered streams received by the receiving means based on the selection information.

The stream of the predetermined category referred to by the main playback path may be a first video stream, and the selection information may further include information defining a combination of a second video stream different from the first video stream and a second audio stream associated with the second video stream.

The stream of the predetermined category referred to by the main playback path may be a first video stream, and the selection information may further include information defining a combination of a second video stream different from the first video stream and first image information associated with the second video stream.

The playback apparatus may further include: first storing means for storing information indicating whether second image information associated with the first video stream is to be displayed or whether the first image information associated with the second video stream is to be displayed; second storing means for storing information indicating which stream from among streams forming the second image information that are selectable by the selection information is to be played back if the second image information is displayed; third storing means for storing information indicating which stream from among streams forming the first image information that are selectable by the selection information is to be played back if the first image information is displayed; and fourth storing means for storing information indicating whether the first image information or the second image information is to be displayed. The reading means may read a stream associated with the first image information or the second image information to be displayed by further referring to the information stored in the first storing means, the second storing means, the third storing means, and the fourth storing means.

The stream of the predetermined category referred to by the main playback path may be a first video stream, and the selection information may further include information defining a combination of a second video stream, which is independent of the first video stream, and text information associated with the second video stream.

At least one type of at least one sub playback path may be defined according to a category of a stream referred to by the corresponding sub playback path, and at least one sub playback path included in the playback management information may be classified into a predetermined type of at least one type defined, and information concerning the classified type of at least one sub playback path may be included in the playback management information.

At least one type defined may include a type in which at least one category of stream is referred to by the corresponding sub playback path.

At least one type defined may include a type in which at least one category of stream is referred to by the corresponding sub playback path and at least one category of stream may be included in a file different from a file including a stream referred to by the main playback path.

At least one type defined may include a type in which at least one category of stream is referred to by the corresponding sub playback path and at least one category of stream may be included in an identical file including a stream referred to by the main playback path.

The playback apparatus may further include display control means for controlling display of a display screen which displays information used for assisting a user in performing an input operation. The stream of the predetermined category referred to by the main playback path may be a first video stream, and if a selection of a second video stream, which is independent of the first video stream, that is played back in synchronization with the first video stream is received by the receiving means, the display control means may control a list of streams that can be played back in association with the second video stream to be displayed on the display screen, and the receiving means may receive a selection of a stream that can be played back in association with the second video stream from the user by referring to the display screen which is controlled to be displayed by the display control means.

A playback method/program according to one aspect of the present invention includes: an obtaining step of obtaining playback management information including a main playback path indicating a position of at least one stream on a time axis, at least one sub playback path indicating a position of at least one stream on the time axis, and selection information for selecting at least one category of stream, which is to be played back, from among at least one stream referred to by the main playback path and for selecting at least one category of stream, which is to be played back, referred to by one of at least one sub playback path; a receiving step of receiving a selection of streams to be played back based on the selection information included in the playback management information obtained by processing in the obtaining step; a reading step of reading, if the selection of streams to be played back is received by processing in the receiving step, by referring to the main playback path, at least one category of stream selected from among at least one stream referred to by the main playback path, and reading, by referring to the corresponding sub playback path, at least one category of stream selected from among at least one stream referred to by at least one sub playback path; and a combining step of combining at least two streams of a predetermined category, if at least two streams are included in at least one category of stream referred to by the main playback path and read by processing in the reading step and in at least one category of stream referred to by the corresponding sub playback path and read by processing in the reading step. The selection information includes information indicating a combination of a first stream of the predetermined category referred to by the main playback path and a second stream of the predetermined category referred to by the corresponding sub playback path, in the processing in the receiving step, the selection of streams to be played back is received based on the selection information, in the processing in the reading step, if the selection of the first stream and the second stream to be played back is received by the processing in the receiving step, the first stream of the predetermined category referred to by the main playback path and the second stream of the predetermined category referred to by the corresponding sub playback path are read, and in the processing in the combining step, the first stream and the second stream of the predetermined category read by the processing in the reading step are combined.

According to the playback apparatus, playback method, and program according to one aspect of the present invention, playback management information including a main playback path indicating a position of at least one stream on a time axis, at least one sub playback path indicating a position of at least one stream on the time axis, and selection information for selecting at least one category of stream, which is to be played back, from among at least one stream referred to by the main playback path and for selecting at least one category of stream, which is to be played back, referred to by one of at least one sub playback path is obtained. A selection of streams to be played back is received based on the selection information included in the obtained playback management information. If the selection of streams to be played back is received, by referring to the main playback path, at least one category of stream selected from among at least one stream referred to by the main playback path is read, and by referring to the corresponding sub playback path, at least one category of stream selected from among at least one stream referred to by at least one sub playback path is read. At least two streams of a predetermined category are combined if at least two streams are included in at least one category of read stream referred to by the main playback path and in at least one category of read stream referred to by the corresponding sub playback path. The selection information includes information indicating a combination of a first stream of the predetermined category referred to by the main playback path and a second stream of the predetermined category referred to by the corresponding sub playback path, in the processing in the receiving step, the selection of streams to be played back is received based on the selection information. If the selection of the first stream and the second stream to be played back is received, the first stream of the predetermined category referred to by the main playback path and the second stream of the predetermined category referred to by the corresponding sub playback path are read. The read first stream and the read second stream of the predetermined category are combined.

A recording medium according to one aspect of the present invention records thereon data including playback management information for managing a playback operation for at least one stream is recorded. The playback management information includes: a main playback path indicating a position of at least one stream on a time axis; at least one sub playback path indicating a position of at least one stream on the time axis; and selection information including information for selecting at least one category of stream, which is to be played back, from among at least one stream referred to by the main playback path and for selecting at least one category of stream, which is to be played back, referred to by one of at least one sub playback path, the information indicating a combination of a first stream of the predetermined category referred to by the main playback path and a second stream of the predetermined category referred to by at least one sub playback path.

A data structure according to one aspect of the present invention includes playback management information for managing a playback operation for at least one stream. The playback management information includes: a main playback path indicating a position of at least one stream on a time axis; at least one sub playback path indicating a position of at least one stream on the time axis; and selection information including information for selecting at least one category of stream, which is to be played back, from among at least one stream referred to by the main playback path and for selecting at least one category of stream, which is to be played back, referred to by one of at least one sub playback path, the information indicating a combination of a first stream of the predetermined category referred to by the main playback path and a second stream of the predetermined category referred to by at least one sub playback path.

The recording medium/data structure according to one aspect of the present invention includes playback management information for managing a playback operation of a first file including at least one stream. More specifically, the playback management information includes: a main playback path indicating a position of at least one stream on a time axis; at least one sub playback path indicating a position of at least one stream on the time axis; and selection information including information for selecting at least one category of stream, which is to be played back, from among at least one stream referred to by the main playback path and for selecting at least one category of stream, which is to be played back, referred to by one of at least one sub playback path, the information indicating a combination of a first stream of the predetermined category referred to by the main playback path and a second stream of the predetermined category referred to by at least one sub playback path.

A manufacturing method for a recording medium according to one aspect of the present invention is a manufacturing method for a recording medium on which data that can be played back by a playback apparatus is recorded. The manufacturing method includes the step of generating data having a data structure including playback management information for managing a playback operation for at least one stream and recording the generated data on the recording medium. The playback management information including a main playback path indicating a position of at least one stream on a time axis, at least one sub playback path indicating a position of at least one stream on the time axis, and selection information for selecting at least one category of stream, which is to be played back, from among at least one stream referred to by the main playback path and for selecting at least one category of stream, which is to be played back, referred to by one of at least one sub playback path, the information indicating a combination of a first stream of the predetermined category referred to by the main playback path and a second stream of the predetermined category referred to by at least one sub playback path.

The manufacturing method for a recording medium according to one aspect of the present invention is a manufacturing method for a recording medium on which data that can be played back by a playback apparatus is recorded. In the manufacturing method, data having a data structure including playback management information for managing a playback operation for at least one stream is generated, and the generated data is recorded on the recording medium. The playback management information includes a main playback path indicating a position of at least one stream on a time axis, at least one sub playback path indicating a position of at least one stream on the time axis, and selection information for selecting at least one category of stream, which is to be played back, from among at least one stream referred to by the main playback path and for selecting at least one category of stream, which is to be played back, referred to by one of at least one sub playback path, the information indicating a combination of a first stream of the predetermined category referred to by the main playback path and a second stream of the predetermined category referred to by at least one sub playback path.

According to the present invention, AV streams can be played back, and in particular, various types of content, such as sound and video, from a stream or a data file different from a main AV stream can be selected.

• [Fig. 1] Fig. 1 illustrates known sound switching.
• [Fig. 2] Fig. 2 illustrates known sound switching.
• [Fig. 3] Fig. 3 illustrates the structure of an MPEG2 program stream.
• [Fig. 4] Fig. 4 illustrates a stream number table indicating audio signals and subtitle signals provided to a user.
• [Fig. 5] Fig. 5 illustrates an example of an application format on a recording medium installed on a playback apparatus to which the present invention is applied.
• [Fig. 6] Fig. 6 illustrates the structure of Main Path and Sub Paths.
• [Fig. 7] Fig. 7 illustrates an example of Main Path and Sub Paths.
• [Fig. 8] Fig. 8 illustrates another example of Main Path and Sub Paths.
• [Fig. 9] Fig. 9 illustrates the syntax of PlayList().
• [Fig. 10] Fig. 10 illustrates the syntax of SubPath().
• [Fig. 11] Fig. 11 illustrates an example of SubPath_type.
• [Fig. 12] Fig. 12 illustrates the syntax of SubPlayItem(i).
• [Fig. 13] Fig. 13 illustrates the syntax of PlayItem().
• [Fig. 14] Fig. 14 illustrates the syntax of first STN_table().
• [Fig. 15] Fig. 15 illustrates a first example of the syntax of stream_entry().
• [Fig. 16] Fig. 16 illustrates the syntax of second STN_table().
• [Fig. 17] Fig. 17 illustrates a second example of the syntax of stream_entry().
• [Fig. 18] Fig. 18 illustrates the syntax of stream_attribute().
• [Fig. 19] Fig. 19 illustrates stream_coding_type.
• [Fig. 20] Fig. 20 illustrates video_format.
• [Fig. 21] Fig. 21 illustrates frame_rate.
• [Fig. 22] Fig. 22 illustrates aspect_ratio.
• [Fig. 23] Fig. 23 illustrates audio_presentation_type.
• [Fig. 24] Fig. 24 illustrates sampling_frequency.
• [Fig. 25] Fig. 25 illustrates Character code.
• [Fig. 26] Fig. 26 illustrates an example of a stream number table indicating the relationship between audio signals and subtitle signals provided to a user.
• [Fig. 27] Fig. 27 is a block diagram illustrating an example of the configuration of a playback apparatus to which the present invention is applied.
• [Fig. 28] Fig. 28 is a block diagram illustrating an example of the configuration of a playback apparatus to which the present invention is applied.
• [Fig. 29] Fig. 29 is a flowchart illustrating playback processing performed by the playback apparatus shown in Fig. 28.
• [Fig. 30] Fig. 30 is a flowchart illustrating playback processing performed by the playback apparatus shown in Fig. 28.
• [Fig. 31] Fig. 31 is a flowchart illustrating playback processing performed by the playback apparatus shown in Fig. 28.
• [Fig. 32] Fig. 32 is a flowchart illustrating processing in response to an instruction to switch sound from a user.
• [Fig. 33] Fig. 33 is a flowchart illustrating details of step S59 in Fig. 32.
• [Fig. 34] Fig. 34 illustrates an example in which two audio streams are mixed and played back.
• [Fig. 35] Fig. 35 illustrates the syntax of third STN_table().
• [Fig. 36] Fig. 36 is a block diagram illustrating another example of the configuration of a playback apparatus to which the present invention is applied.
• [Fig. 37] Fig. 37 illustrates the syntax of fourth STN_table() defining combinations of audio streams #1 and audio streams #2.
• [Fig. 38] Fig. 38 illustrates an example of the bitmap syntax of Combination_of_Primary_and_Secondary.
• [Fig. 39] Fig. 39 illustrates a stream number table indicating audio signals and subtitle signals provided to a user.
• [Fig. 40] Fig. 40 illustrates another example in which two audio streams are mixed and played back.
• [Fig. 41] Fig. 41 illustrates the syntax of fifth STN_table() defining combinations of audio streams #1 and audio streams #2.
• [Fig. 42] Fig. 42 illustrates the syntax of sixth STN_table() defining combinations of audio streams #1 and audio streams #2.
• [Fig. 43] Fig. 43 illustrates a picture-in-picture technique.
• [Fig. 44] Fig. 44 illustrates the syntax of sixth STN_table() defining combinations of secondary video streams, primary audio streams, secondary audio streams, and subtitle streams that can be played back together with primary video streams.
• [Fig. 45] Fig. 45 illustrates the syntax of seventh STN_table() defining combinations of secondary video streams, primary audio streams, secondary audio streams, and subtitle streams that can be played back together with primary video streams.
• [Fig. 46] Fig. 46 illustrates a stream number table indicating combinations of audio signals, video signals, and subtitle signals provided to a user.
• [Fig. 47] Fig. 47 illustrates a user selection operation for selecting a combination of a secondary video stream, a primary audio stream, a secondary audio stream, and a subtitle stream that can be played back together with a primary video stream.
• [Fig. 48] Fig. 48 illustrates a PlayList generated by using SubPath_type shown in Fig. 11 to implement the picture-in-picture technique.
• [Fig. 49] Fig. 49 illustrates another example of SubPath_type different from SubPath_type shown in Fig. 11.
• [Fig. 50] Fig. 50 illustrates a third example of stream_entry().
• [Fig. 51] Fig. 51 illustrates an example of PlayList generated by using SubPath_type shown in Fig. 49 to implement the picture-in-picture technique.
• [Fig. 52] Fig. 52 illustrates another example of PlayList generated by using SubPath_type shown in Fig. 49 to implement the picture-in-picture technique.
• [Fig. 53] Fig. 53 is a block diagram illustrating an example of the configuration of a playback apparatus to which the present invention is applied, which is different from the example shown in Fig. 36.
• [Fig. 54] Fig. 54 is a flowchart illustrating secondary video switching processing performed by the playback apparatus shown in Fig. 53.
• [Fig. 55] Fig. 55 is a flowchart illustrating the secondary video switching processing performed by the playback apparatus shown in Fig. 53.
• [Fig. 56] Fig. 56 illustrates the types of application_type.
• [Fig. 57] Fig. 57 illustrates a Sub Path restriction technique, and more specifically, Fig. 57 illustrates Sub Paths that can be provided for PlayList according to the types of PlayList.
• [Fig. 58] Fig. 58 illustrates a Sub Path restriction technique, and more specifically, Fig. 58 illustrates the relationships between the number of PlayItems disposed in PlayList and the number of Sub Paths.
• [Fig. 59] Fig. 59 illustrates a specific example of PlayList generated by the Sub Path restriction technique shown in Figs. 57 through 59.
• [Fig. 60] Fig. 60 illustrates a specific example of PlayList generated by the Sub Path restriction technique shown in Figs. 57 through 59.
• [Fig. 61] Fig. 61 illustrates a specific example of PlayList generated by the Sub Path restriction technique shown in Figs. 57 through 59.
• [Fig. 62] Fig. 62 illustrates a specific example of PlayList generated by the Sub Path restriction technique shown in Figs. 57 through 59.
• [Fig. 63] Fig. 63 is a flowchart illustrating playback processing for PlayList having application_type=3 in the playback apparatus shown in Fig. 53.
• [Fig. 64] Fig. 64 is a flowchart illustrating the playback processing for PlayList having application_type=3 in the playback apparatus shown in Fig. 53.
• [Fig. 65] Fig. 65 is a flowchart illustrating playback processing for a PlayList having application type=1 or 2 in the playback apparatus shown in Fig. 53.
• [Fig. 66] Fig. 66 is a flowchart illustrating the playback processing for a PlayList having application_type=1 or 2 in the playback apparatus shown in Fig. 53.
• [Fig. 67] Fig. 67 is a flowchart illustrating the playback processing for a PlayList having application_type=1 or 2 in the playback apparatus shown in Fig. 53.
• [Fig. 68] Fig. 68 is a flowchart illustrating the playback processing for a PlayList having application_type=1 or 2 in the playback apparatus shown in Fig. 53.
• [Fig. 69] Fig. 69 illustrates the syntax of eighth STN_table() defining combinations of audio streams #1 and audio streams #2, video streams #2 and audio streams #1, and video streams #2 and video stream #2 text subtitle streams.
• [Fig. 70] Fig. 70 illustrates the syntax of eighth STN_table() defining combinations of audio streams #1 and audio streams #2, video streams #2 and audio streams #1, and video streams #2 and video stream #2 text subtitle streams.
• [Fig. 71] Fig. 71 illustrates registers.
• [Fig. 72] Fig. 72 illustrates an example in which a primary PG TextST stream is played back when a PiP display operation is executed.
• [Fig. 73] Fig. 73 illustrates an example in which a picture-in-picture application PG TextST stream is played back when a picture-in-picture display operation is executed.
• [Fig. 74] Fig. 74 is a flowchart illustrating PiP (picture-in-picture) application PG TextST playback processing.
• [Fig. 75] Fig. 75 is a flowchart illustrating PiP PG TextST stream display processing.
• [Fig. 76] Fig. 76 illustrates the syntax of ninth STN_table() defining a larger number of combinations.
• [Fig. 77] Fig. 77 illustrates the syntax of ninth STN_table() defining a larger number of combinations.
• [Fig. 78] Fig. 78 illustrates the syntax of ninth STN_table() defining a larger number of combinations.
• [Fig. 79] Fig. 79 illustrates registers.
• [Fig. 80] Fig. 80 illustrates an example in which a primary audio stream and a secondary audio stream are mixed and played back when a picture-in-picture display operation is executed.
• [Fig. 81] Fig. 81 illustrates an example in which a picture-in-picture application primary audio stream is played back when a picture-in-picture display operation is executed.
• [Fig. 82] Fig. 82 is a flowchart illustrating PiP application audio stream playback processing 1.
• [Fig. 83] Fig. 83 illustrates the syntax of tenth STN_table() defining a larger number of combinations.
• [Fig. 84] Fig. 84 illustrates the syntax of tenth STN_table() defining a larger number of combinations.
• [Fig. 85] Fig. 85 illustrates the syntax of tenth STN_table() defining a larger number of combinations.
• [Fig. 86] Fig. 86 illustrates an example in which a picture-in-picture application primary audio stream and a secondary audio stream are mixed and played back when a picture-in-picture display operation is executed.
• [Fig. 87] Fig. 87 is a flowchart illustrating PiP application audio stream playback processing 2.
• [Fig. 88] Fig. 88 illustrates a manufacturing method for a recording medium on which data to be played back by an playback apparatus.
• [Fig. 89] Fig. 89 illustrates the manufacturing method for a recording medium on which data to be played back by an playback apparatus.
• [Fig. 90] Fig. 90 illustrates the configuration of a personal computer.

1 playback apparatus, 11 recording medium, 21 controller, 22 optical disc drive, 23 memory, 24 local storage, 26 AV decoder, 32 switch, 51 through 54 buffers, 55, 56 PID filters, 57 through 59 switches, 71 background decoder, 72 video decoder/first video decoder, 73 presentation graphics decoder, 74 interactive graphics decoder, 75 audio decoder, 76 Text-ST composition, 77 switch, 91 background plane generator, 92 video plane generator/first video plane generator, 93 presentation graphics plane generator, 94 interactive graphics plane generator, 95 buffer, 96 video data processor, 97 mixing processor, 101 mixing processor, 141 playback apparatus, 151 AV decoder, 161 PID filter, 162 switch, 401 playback apparatus, 403 AV decoder, 411 PID filter, 412 second video decoder, 413 second video plane generator, 414 preload buffer

An embodiment of the present invention is described below with reference to the accompanying drawings.

Fig. 5 illustrates an example of an application format on a local storage (for example, a hard disk or a memory) built in a playback apparatus 1 (which is discussed below with reference to Figs. 27 and 28) to which the present invention is applied or on a recording medium to be installed in the playback apparatus 1. The recording medium is an optical disc, a magnetic disk, or a semiconductor memory, which is discussed below.

The application format has two layers, i.e., a PlayList layer and a Clip layer, for managing AV (Audio Visual) streams. In this case, a pair of one AV stream and one item of clip information associated with the AV stream are considered to be as one object, which is referred to as a "clip". An AV stream is also referred to as an " AV stream file". The AV stream file may include, not only sound data and video data, but also various stream files played back together with the sound data and the video data. Clip information is also referred to as a "clip information file".

Generally, files used in computers are handled as byte strings. On the other hand, content of AV stream files is expanded onto a time axis, and access points in clips are mainly specified by PlayLists by using time stamps. That is, it can be said that the PlayLists and clips form two layers for managing AV streams.

If access points in clips are indicated by PlayLists by using time stamps, a Clip Information file is used for finding, from the time stamps, information concerning an address at which decoding in an AV stream file is started.

A PlayList is a set of playback zones of an AV stream. One playback zone in an AV stream is referred to as a "PlayItem", which is indicated by a pair of an IN point (playback start point) and an OUT point (playback end point) on the time axis. Accordingly, a PlayList has one or a plurality of Playltems, as shown in Fig. 5.

In Fig. 5, the first PlayList from the left has two PlayItems, which refer to the first half and the second half of the AV stream contained in the clip at the left side in Fig. 5. The second PlayList from the left has one PlayItem, which refers to the entirety of the AV stream contained in the clip at the right side. The third PlayList from the left has two PlayItems, which refer to a certain portion of the AV stream contained in the clip at the left side and a certain portion of the AV stream contained in the clip at the right side.

If a disc navigation program shown in Fig. 5 designates the left PlayItem contained in the first PlayList from the left as information concerning the current playback position, the first half of the AV stream contained in the left clip, which is referred to by the designated PlayItem, is played back. In this manner, the PlayLists are used as playback management information for managing the playback operation of AV stream files.

The disc navigation program has a function of controlling the playback order specified in PlayLists and interactive playback operations by using PlayLists. The disc navigation program also has a function of displaying a menu screen for allowing a user to give instructions to perform various types of playback operations. The disc navigation program is described in a programming language, for example, Java^™, and is recorded on a recording medium.

In this embodiment, a playback path including at least one PlayItem (sequential PlayItems) in a PlayList is referred to as a "Main Path", and a playback path including at least one SubPlayItem (may be sequential or non-sequential PlayItems) disposed in parallel with the Main Path in a PlayList is referred to as a "Sub Path". That is, the application format on the local storage built in the playback apparatus 1 (discussed below with reference to Figs. 27 and 28) or on a recording medium installed in the playback apparatus 1 has at least one Sub Path, which is played back in association with the Main Path, in a PlayList.

Fig. 6 illustrates the structure of the Main Path and Sub Paths. A PlayList is allowed to have the single Main Path and at least one Sub Path. The Main Path includes at least one PlayItem, and one Sub Path includes at least one SubPlayItem.

The PlayList shown in Fig. 6 has one Main Path including three Playltems and three Sub Paths. The PlayItems forming the Main Path are provided with IDs (Identifications) from the head. More specifically, the Main Path includes Playltems, such as Playltem_id=0, PlayItem_ id=1, and PlayItem_id=2. The Sub Paths are also provided with IDs from the head, such as, Subpath_id=0, Subpath_id=1, and Subpath_id-2. Subpath_id=0 has one SubPlayItem, Subpath_id=1 has two SubPlayItems, and SubPath_id=2 has one SubPlayItem.

The stream referred to by the SubPlayItem contained in Sub Path_id=0 is, for example, dubbed Japanese movie sound, and can be played back instead of the audio stream contained in the AV stream file referred to by the Mainpath. The stream referred to by the SubPlayItems contained in Sub Path_id=1 is, for example, Director's Cut, and can be inserted into a predetermined portion of the AV stream file referred to by the Main Path as the director's commentaries.

A clip AV stream file referred to by one PlayItem includes at least video stream data (main image data). The clip AV stream file may also include at least one audio stream, which is played back simultaneously with (in synchronization with) the video stream (main image data) which is also contained in the clip AV stream file. The clip AV stream file may also include at least one bitmap subtitle stream file (Presentation Graphics stream) which is played back in synchronization with the video stream which is also contained in the clip AV stream file. The clip AV stream file may also include at least one interactive graphics stream file which is played back in synchronization with the video stream which is also contained in the clip AV stream file. The video stream contained in the clip AV stream file and the audio stream, bitmap subtitle stream, or interactive graphics stream, which is played back in synchronization with the video stream, are multiplexed. In other words, a clip AV stream file referred to by one PlayItem includes video stream data and at least 0 audio stream data, at least 0 bitmap subtitle stream data, or at least 0 interactive graphics stream data, which is played back in synchronization with the video stream data, such that they are multiplexed into the clip AV stream file.

That is, a clip AV stream file referred to by one PlayItem includes a plurality of types of streams, such as a video stream, an audio stream, a bitmap subtitle stream, or an interactive graphics stream.

A SubPlayItem refers to, for example, audio stream data or subtitle data contained in a stream file different from the clip AV stream file referred to by the Playltem.

When playing back a PlayList including only a Main Path, the user can select sound and subtitles only from audio streams and sub-picture streams multiplexed into a clip referred to by that Main Path. In contrast, when playing back a PlayList including a Main Path and a Sub Path, the user can refer to audio streams and sub-picture streams multiplexed into a clip referred to by the SubPlayItem in addition to audio streams and sub-picture streams multiplexed into a clip AV stream file referred to by the Main Path.

As discussed above, a plurality of Sub Paths are included in one PlayList, and a SubPlayItem of each Sub Path refers to the corresponding streams. Accordingly, AV streams having high extensibility and high flexibility can be provided. That is, SubPlayItems can be added afterwards.

Fig. 7 illustrates an example of a Main Path and an example of a Sub Path. In Fig. 7, an audio playback path played back simultaneously with (in synchronization with) the Main Path is indicated by using the Sub Path.

The PlayList shown in Fig. 7 includes one PlayItem, i.e., PlayItem_id=0, as the Main Path, and one SubPlayItem as the Sub Path. PlayItem(), which is PlayItem_id=0, refers to the main clip AV stream. SubPlayItem() includes the following data. SubPlayItem() includes Clip_Information_file_name for specifying the clip referred to by the Sub Path in the PlayList. In the example in Fig. 7, the SubPlayItem refers to an auxiliary audio stream of SubClip_entry_id=0. SubPlayItem() also includes SubPlayItem_IN_time and SubPlayItem_OUT_time for specifying the playback zone of the Sub Path contained in the clip (in this case, the auxiliary audio stream). SubPlayItem() also includes sync_PlayItem_id and sync_start PTS of Playltem for specifying the time at which the playback operation of the Sub Path is started on the time axis of the Main Path. In Fig. 7, sync_PlayItem_id=0 and sync_start_PTS_of_PlayItem=t1. With this information, the time t1 at which the playback operation of the Sub Path is started on the time axis of PlayItem=0 of the Main Path can be specified. That is, in the example in Fig. 7, the playback start time of the Main Path and the playback start time of the Sub Path is the same, i.e., t1.

The clip AV audio stream referred to by the Sub Path should not include STC non-sequential time points (system time base non-sequential points). The clip audio sample clock used for the Sub Path is locked to the audio sample clock used for the Main Path.

In other words, SubPlayItem() includes information for specifying the clip referred to by the Sub Path, information for specifying the playback zone of the Sub Path, and information for specifying the time at which the playback operation of the Sub Path is started on the time axis of the Main Path. Since the clip AV stream used for the Sub Path does not include STC, the user can refer to a clip AV audio stream different from the clip AV stream (main AV stream to be played back) referred to by the Main Path on the basis of the information included in SubPlayItem() (information for specifying the clip referred to by the Sub Path, information for specifying the playback zone of the Sub Path, and information for specifying the time at which the playback operation of the Sub Path is started on the time axis of the Main Path), and plays back the clip AV audio stream.

As stated above, the PlayItem and the SubPlayItem individually manage clip AV stream files. The clip AV stream file (main AV stream file) managed by the PlayItem is different from the clip AV stream file managed by the SubPlayItem.

In a manner similar to the example shown in Fig. 7, a subtitle stream playback path played back simultaneously with the Main Path may be indicated by using a Sub Path.

Fig. 8 illustrates another example of a Main Path and another example of Sub Paths. In Fig. 8, an audio playback path played back AV-simultaneously with the Main Path is indicated by using a Sub Path. The main clip AV stream file referred to by the PlayItem of the Main Path is similar to that in Fig. 7, and an explanation thereof is thus omitted.

It is now assumed that the clip AV stream referred to by the Main Path is movie content (AV content), and the auxiliary audio stream (clip) referred to by the audio playback path of the Sub Path is director's commentaries on that movie, and then, the auxiliary audio stream referred to by the Sub Path is mixed into (overlaps with) the clip AV audio stream referred to by the Main Path. In this case, the configuration shown in Fig. 8 can be used. More specifically, the configuration shown in Fig. 8 can be used when the user inputs an instruction to listen to the director's commentaries on the movie into a player while watching the movie, in other words, when the auxiliary audio stream referred to by the Sub Path is mixed into the sound of the clip AV stream referred to by the Main Path.

In Fig. 8, three PlayItems, i.e., PlayItem_id=0, 1, and 2 are disposed in the Main Path, and two SubPlayItems are disposed in the Sub Path (Subpath_id=0). The SubPlayItem (discussed below with reference to Fig. 12) called by the Sub Path (Subpath_id=0) (discussed below with reference to Fig. 10) includes SubPlayItem_IN-time and SubPlayItem_OUT_time for specifying the playback zones of the auxiliary audio streams (English auxiliary audio stream clip having SubClip_entry_id=0 and Japanese auxiliary audio stream clip having SubClip_entry_id=1) of the Sub Path.

Comparing the example in Fig. 8 with that in Fig. 7, it can be seen that, in the example in Fig. 8, a SubPlayItem can refer to the auxiliary audio stream (English or Japanese audio stream) having SubClip_entry_id=0 or SubClip_entry_id=1. That is, the use of the SubPlayItem makes it possible to refer to a plurality of audio streams, and when playing back the SubPlayItem, an audio stream file can be selected from a plurality of audio stream files, i.e., in the example in Fig. 8, the English audio stream file or the Japanese audio stream file is selected. More specifically, one of SubClip_entry_id=0 and SubClip entry id=1 is selected based on, for example, an instruction from the user, and the auxiliary audio stream referred to by the selected ID is played back. If an instruction to play back the auxiliary audio stream together with the audio stream referred to by the Main Path is provided, a first audio stream referred to by the Main Path and a second audio stream referred to by the Sub Path are mixed and played back. It is possible that two audio streams referred to by the two Sub Paths be mixed and played back. This is discussed later.

The data structure (syntax) that implements the structure of the Main Path and Sub Paths discussed with reference to Figs. 6 through 8 is as follows.

Fig. 9 illustrates the syntax of PlayList().

The length is a 32-bit unsigned integer indicating the number of bytes from immediately after the length field to the end of PlayList(), i.e., a field indicating the number of bytes from reserved_for_future_use to the end of PlayList(). After the length, the 16-bit reserved_for_future_use field follows. The number_of_PlayItems is a 16-bit field indicating the number of PlayItems contained in the PlayList. In the case of the example in Fig. 6, the number of PlayItems is three, and the numeric value is assigned to the PlayItems as PlayItem_id from 0 in the order in which PlayItem() appears in the PlayList. For example, PlayItem_id=0, 1, 2 are assigned, as shown in Figs. 6 and 8.

The number_of_SubPath is a 16-bit field indicating the number of Sub Paths (number of entries) contained in the PlayList. In the case of the example in Fig. 6, the number of Sub Paths is three, and the numeric value is assigned to the SubPlayItems as SubPath_id from 0 in the order in which SubPath() appears in the PlayList. For example, Sub Path_id=0, 1, 2 are assigned, as shown in Fig. 6. Then, in the subsequent FOR statement, the PlayItems are referred to for the same number of times as the number of PlayItems, and the Sub Paths are referred to for the same number of times as the number of Sub Paths.

Fig. 10 illustrates the syntax of SubPath().

The length is a 32-bit unsigned integer indicating the number of bytes from immediately after the length field to the end of PlayList(), i.e., a field indicating the number of bytes from reserved_for_future_use to the end of PlayList(). After the length, the 16-bit reserved_for_future_use field follows. The SubPath_type is an 8-bit field indicating the type of application of the Sub Path, i.e., the type of Sub Path, such as audio, bitmap subtitle, or text subtitle. A first example of the SubPath type is discussed below with reference to Fig. 11. After the SubPath_type, the 15-bit reserved_for_future_use field follows. The is_repeat_SubPath field is a one-bit field indicating the playback method for the Sub Path, and more specifically, indicating whether the Sub Path is played back repeatedly or only once while playing back the Main Path. This field is used when, for example, the playback timing of the stream contained in the clip specified by the Sub Path is different from that of the main AV stream (when, for example, the Main Path is a still-image slideshow and the audio Sub Path is background music (BGM) of the Main Path). After the is_repeat_SubPath field, the 8-bit reserved_for_future_use follows. The number_of SubPlayItems is an 8-bit field indicating the number of SubPlayItems (number of entries) contained in one Sub Path. For example, the number of SubPlayItems of SubPath_id=0 in Fig. 6 is one, and the number of SubPlayItems of SubPath_id=1 is 2. In the subsequent FOR statement, the SubPlayItems are referred to for the same number of times as the number of SubPlayItems.

Fig. 11 illustrates a first example of the SubPath_type (the type of Sub Path). The types of Sub Paths can be defined, for example, as shown in Fig. 11.

In Fig. 11, SubPath_type=0, 1 are reserved. The SubPath_type=2 is the Audio presentation path of the Browsable slideshow. In SubPath_type=2, the audio presentation path referred to by the Sub Path is not synchronized with the Main Path referred to by the PlayItems in the PlayList.

The Sub Path_type=3 is the Interactive graphics presentation menu. In SubPath_type=3, the interactive graphics presentation menu referred to by the Sub Path is not synchronized with the Main Path referred to by the PlayItems in the PlayList.

The SubPath_type=4 is the Text subtitle presentation path. In SubPath_type=4, the text subtitle presentation path referred to by the Sub Path is synchronized with the Main Path referred to by the PlayItems in the PlayList.

The SubPath_type=5 is the second Audio presentation path (which is the path for referring to the second audio stream). In SubPath_type=5, the second audio presentation path referred to by the Sub Path is synchronized with the Main Path referred to by the PlayItems in the PlayList. The second audio stream referred to by that Sub Path is, for example, director's commentaries (sound) on a movie. For the Sub Path represented by Subpath_id=0 in Fig. 8, the SubPath_type is SubPath_type=5 in Fig. 10.

The SubPath_type=6 is the second video presentation path (which is the path for referring to the second video stream). In SubPath_type=6, the second video presentation path referred to by the Sub Path is synchronized with the Main Path referred to by the PlayItems in the PlayList. The second video stream referred to by that Sub Path is, for example, director's commentaries (moving pictures) on a movie. When describing processing performed by the playback apparatus 1 with reference to in Figs. 27 and 28, only audio synthesizing is discussed. However, moving pictures can also be synthesized in a configuration similar to that shown in Figs. 27 and 28, which is discussed in detail later.

The SubPath_type=7 through 255 are reserved.

Fig. 12 illustrates the syntax of SubPlayltem(i).

The length is a 16-bit unsigned integer indicating the number of bytes from immediately after the length field to the end of SubPlayItem ().

In Fig. 12, the syntax is divided into two portions, and more specifically, a portion where the SubPlayItem refers to one clip and a portion where the SubPlayItem refers to a plurality of clips are shown.

The portion where the SubPlayItem refers to one clip is discussed first.

The SubPlayItem() includes Clip Information_file_name[0] for specifying the clip, Clip_codec_identifier[0] for specifying the codec method for the clip, reserved_for_future_use, is_multi_Clip_entries, which is a flag indicating whether multi-clips are registered, and ref_to_STC_id[0], which is information concerning the STC non-sequential points (non-sequential points of the system time base). If the flag of is multi Clip entries is ON, the syntax of the portion where SubPlayItem() refers to a plurality of clips is checked. The SubPlayItem() also includes SubPlayItem IN_time and SubPlayItem_OUT_time for specifying the playback zone of the Sub Path contained in the clip, and sync_PlayItem_id and sync_start_PTS_of_PlayItem for specifying the playback start time at which the playback operation of the Sub Path is started on the time axis of the Main Path. The sync_PlayItem_id and sync_start_PTS_of_PlayItem are used when the playback timing of the main AV stream is the same as that of the stream contained in the file referred to by the Sub Path, as shown in Figs. 7 and 8, but are not used when the playback timing of the main AV stream is different from the stream contained in the file referred to by the Sub Path (for example, still-image slideshow (Main Path) is not synchronized with BGM (Sub Path) for the slideshow). The SubPlayItem_IN_time, SubPlayItem OUT time, sync_PlayItem_id, and sync_start_PTS of_PlayItem are used in common for the clip referred to by SubPlayItem ().

Next, the portion where the SubPlayItem refers to a plurality of clips (if (is_multi_Clip_entries==1b) is discussed. More specifically, the case where the SubPlayItem refers to a plurality of clips, as shown in Fig. 8, is discussed.

The num_of_Clip_entries indicates the number of clips, and designates clips other than those having Clip_Information_file_name[0] [SubClip_entiry_id]. That is, num_of_Clip_entries designates clips, such as those having Clip_Information_file_name[1], Clip_Information_file_name[2], and so on, other than those having Clip_Information_file_name[0]. The SubPlayItem() also includes Clip_codec_identifier[SubClip_entry_id] for specifying the codec method for the clip, ref to STC id[SubClip_entry_id], which is information concerning the STC non-sequential points (non-sequential points of the system time base), and reserved_for_future_use.

The SubPla tem_IN_time, SubPlayItem_OUT_time, sync_PlayItem_id, and sync_start PTS_of PlayItem are used in common for the clips referred to by SubPlayItem(). In the example in Fig. 8, SubPlayItem_IN_time, SubPlayItem_OOT_time, sync_PlayItem_id, and sync_start_PTS_of_PlayItem are used in common for SubClip_entry_id=0 and SubClip_entry_id=1. The text-based subtitle for the selected SubClip_entry_id is played back based on the SubPlayItem_IN_time, SubPlayItem_OUT_time, sync_PlayItem_id, and sync_start PTS_of_PlayItem.

The numeric value is sequentially assigned to SubClip_entry_id from 1 in the order in which Clip_Information_file_name[SubClip_entry_id] in the SubPlayltem() appears. The SubClip_entry_id of Clip_Information file name[0] is 0.

Fig. 13 illustrates the syntax of Playltem().

The length is a 16-bit unsigned integer indicating the number of bytes from immediately after the length field to the end of PlayItem(). Clip_Information_file name[0] is a field for specifying the clip referred to by PlayItem(). In the example in Fig. 7, the main clip AV stream file is referred to by Clip_Information_file_name[0]. The PlayItem() also includes Clip_codec_identifier[0] specifying the codec method for the clip, reserved_for_future_use, is_multi_angle, connection_condition, and ref_to_STC_id[0], which is information concerning the STC non-sequential points (non-sequential time points of the system time base). The PlayItem() also includes IN_time and OUT_time for specifying the playback zone of the PlayItem in the clip. In the example in Fig. 7, IN_time and OUT_time specify the playback zone of the main clip AV stream file. The Playltem() also includes UO_mask_table(), Playltem_random_access_mode, and still_mode. A description of a case where is_multi_angle indicates a plurality of angles is not given here, since such a case is not directly related to the present invention.

The STN_table() provides a mechanism for allowing a user, if the target PlayItem and at least one Sub Path to be played back in association with the target PlayItem are provided, to select from the streams contained in the clip referred to by the PlayItem and the clips referred to by at least Sub Path when the user switches sound or subtitles.

Fig. 14 illustrates an example of the syntax of STN_table() (first STN_table()). The first STN_table() is set as an attribute of PlayItem(). The first STN_table() shown in Fig. 14 provides a mechanism for allowing the user to select a combination of the clip referred to by the PlayItem and the clip referred to by the SubPath.

The length is a 16-bit unsigned integer indicating the number of bytes from immediately after the length field to the end of STN_table(). After the length, a 16-bit reserved_for_future_use follows. The number_of_video_stream_entries indicates the number of streams provided with video_stream_id entered (registered) in STN_table(). The video_stream_id is information for identifying the video streams. The video_stream_number is the video stream number that can be seen by the user when switching video. The number_of_audio_stream_entries indicates the number of streams provided with audio_stream_id entered (registered) in STN_table(). The audio_stream_id is information for identifying the audio streams. The audio stream number is the audio stream number that can be seen by the user when switching sound.

The number of PG txtST stream entries indicates the number of streams provided with PG_txtST_stream_id entered in STN_table(). In STN_table() shown in Fig. 14, streams (PG, Presentation Graphics streams) in which bitmap subtitles, such as DVD sub-pictures, are run-length coded, and text subtitle files (txtST) are entered. The PG_txtST_stream_id is information for identifying the subtitle streams, and PG_txtST_stream number is the subtitle stream number (text subtitle stream number) that can be seen by the user when switching subtitles.

The number_of_IG_stream_entries indicates the number of streams provided with IG_stream_id entered in STN_table(). In STN_table() shown in Fig. 14, interactive graphics streams are entered. IG_stream_id is information for identifying the interactive graphics streams. The IG_stream_number is the graphics stream number that can be seen when switching graphics.

The syntax of stream_entry() is discussed below with reference to Fig. 15.

The length is an 8-bit unsigned integer indicating the number of bytes from immediately after the length field to the end of stream_entry (). The type is an 8-bit field indicating the type of information required for uniquely specifying the stream provided with the above-described stream number.

If type=1 or type=3, a 16-bit packet ID (PID) is designated for specifying one elementary stream from a plurality of elementary streams multiplexed into the main clip referred to by the PlayItem. The ref_to_stream_PID_of_mainClip indicates this PID. That is, if type=1 or type=3, the stream can be determined only by specifying the PID in the main clip AV stream file.

If type=2 or type=3, to specify one elementary stream from a plurality of elementary streams multiplexed into one clip referred to by the Sub Path including a plurality of clips, SubPath_id of the Sub Path, Clip_id, and packet ID (PID) are designated. The ref_to SubPath_id indicates the SubPath_id, the ref_to_SubClip_entry_id indicates the Clip_id, and the ref_to_stream_PID_of SubClip indicates the PID. This field is used when a plurality of clips are referred to by a SubPlayItem and when a plurality of elementary streams are referred to by each clip.

More specifically, if type=3, when the packet ID (PID) for specifying one elementary stream from a plurality of elementary streams multiplexed into the main clip referred to by the PlayItem is designated and when the Sub Path refers to a plurality of clips at the same time and a plurality of elementary streams are multiplexed into each clip, two streams can be entered by SubPath_id, Clip_id, and packet ID (PID) for specifying one elementary stream from a plurality of elementary streams multiplexed into one sub-clip. For example, two audio streams, i.e., an audio stream referred to by the Main Path and an audio stream referred to by the Sub Path, can be entered. Hereinafter, an audio stream referred to by the Main Path is also referred to as a "first audio stream", and an audio stream referred to by the Sub Path is also referred to as a "second audio stream".

If a PlayItem and at least one Sub Path played back in association with the PlayItem are provided, type (three types) can be used for specifying one elementary stream from the clip referred to by the PlayItem and the clips referred to by at least one Sub Path. The type=1 indicates the clip (main clip) referred to by the Main Path, and type=2 or type=3 indicates the clip (sub clip) referred to by the Sub Path.

If type=3, an audio stream (first audio stream) of a clip AV stream file referred to by the Main Path and an audio stream (second audio stream) referred to by the Sub Path can be specified and entered. For example, the first audio stream and the second audio stream can be combined (mixed) and played back.

Referring back to a description of the first STN_table () in Fig. 14, in the FOR loop of the video stream ID (video_stream_id), video_stream_id is assigned from 0 to a video elementary stream specified for each stream entry(). Instead of the video stream ID (video_stream_id), the video stream number (video_stream_number) is used, in which case, the number is assigned from 1. That is, the number obtained by adding one to video_stream_id is the video_stream_number. The number is assigned from 1 since video_stream_number is the video stream number that can be seen by the user when switching video.

Similarly, in the FOR loop of the audio stream ID (audio_stream_id), audio_stream_id is assigned from 0 to an audio elementary stream specified for each stream_entry(). As in the video stream, instead of the audio stream ID (audio_stream_id), the audio stream number (audio_stream_number) is used, in which case, the number is assigned from 1. That is, the number obtained by adding one to audio_stream_id is the audio stream number. The number is assigned from 1 since audio_stream_number is the audio stream number that can be seen by the user when switching sound.

Similarly, in the FOR loop of the subtitle stream ID (PG txtST stream id), PG_txtST_stream_id is assigned from 0 to a bitmap subtitle or text subtitle elementary stream specified for each stream_entry(). As in the video stream, instead of the subtitle stream ID (PG_txtST_stream_id), the subtitle stream number (PG_txtST_stream_number) is used, in which case, the number is assigned from 1. That is, the number obtained by adding one to PG_txtST_stream_id is the PG_txtST stream_number. The number is assigned from 1 since PG_txtST_stream_number is the bitmap subtitle or text subtitle stream number that can be seen by the user when switching subtitles.

Similarly, in the FOR loop of the graphics stream ID (IG_stream_id), IG_stream_id is assigned from 0 to an interactive graphics elementary stream specified for each stream_entry(). As in the video stream, instead of the graphics stream ID (IG_stream_id), the graphics stream number (IG_stream_number) is used, in which case, the number is assigned from 1. That is, the number obtained by adding one to IG_stream_id is the IG_stream_number. The number is assigned from 1 since IG_stream_number is the graphics stream number that can be seen by the user when switching graphics.

The stream_attribute() in the first STN_table() shown in Fig. 14 is as follows. In stream_attribute(), attribute information concerning the stream specified by stream_entry() is recorded.

The stream_attribute() in the FOR loop of the video stream ID (video_stream_id) provides stream attribute information concerning one video elementary stream specified by each stream_entry(). That is, in stream_attribute(), stream attribute information concerning one video elementary stream specified by each stream_entry() is indicated.

The stream_attributes () in the FOR loop of the audio stream ID (audio_stream_id) provides stream attribute information concerning at least one audio elementary stream specified by each stream_entry (). That is, in stream_attributes (), stream attribute information concerning at least one audio elementary stream specified by each stream_entry() is indicated. More specifically, in stream_entry() shown in Fig. 15, if type=1 or type=2, there is only one audio elementary stream specified by this stream_entry(). Accordingly, stream_attributes() provides stream attribute information concerning that audio elementary stream. If type=3, there are two audio elementary streams (i.e., a first audio stream referred to by the Main Path and a second audio stream referred to by a Sub Path) specified by this stream_entry(). Accordingly, stream_attributes () provides stream attribute information concerning the two audio elementary streams.

In this manner, if type=3, i.e., if audio streams are referred to by both the Main Path and a Sub Path, two stream_attributes elements can be inserted. That is, in the FOR statement for the audio streams, if type=3 in stream_entry(), two stream_attributes() elements are entered.

More specifically, in the first STN_table() in Fig. 14, stream_attributes() is defined for each stream_entry() of the audio stream ID (audio_stream_id). If type=3, two stream_attributes() elements are assigned to stream_entry(). Accordingly, since two stream_attributes() elements of the audio streams referred to by the Main Path and a Sub Path are entered for one audio_stream_id, two audio streams which are simultaneously played back can be entered.

The stream_attribute() in the FOR loop of the subtitle stream ID (PG_txtST_stream_id) provides stream attribute information concerning one bitmap subtitle elementary stream or one text subtitle elementary stream specified by each stream_entry (). That is, in stream_attribute (), stream attribute information concerning one bitmap subtitle elementary stream or one text subtitle elementary stream specified by each stream_entry() is indicated.

Similarly, the stream_attribute() in the FOR loop of the graphics stream ID (IG_stream_id) provides stream attribute information concerning one interactive graphics elementary stream specified by each stream_entry(). That is, in stream_attribute(), stream attribute information concerning one interactive graphics elementary stream specified by each stream_entry() is indicated.

An example in which two audio streams, i.e., an audio stream referred to by the Main Path and an audio stream referred to by a Sub Path, are entered has been discussed with reference to Figs. 14 and 15. The syntax of second STN_table() and the syntax of second Stream_entry() that allow two audio streams referred to by two Sub Paths to be entered are shown in Figs. 16 and 17, respectively. In Figs. 16 and 17, elements corresponding to those in Figs. 14 and 15 are not explained for avoiding repetitions.

Fig. 16 illustrates the syntax of second STN_table(). The second STN_table() is set as an attribute of the PlayItem. Fig. 17 illustrates the syntax of second Stream_entry().

In Fig. 17, if type=1, to specify one elementary stream from a plurality of elementary streams multiplexed into the main clip referred to by the PlayItem, a 16-bit packet ID (PID) is designated. The ref_to_stream_PID_of_mainClip indicates this PID. That is, if type=1, the stream is determined only by specifying the PID in the main clip AV stream file.

If type=2, to specify one elementary stream from a plurality of elementary streams multiplexed into one clip referred to by a Sub Path including a plurality of clips, the SubPath_id, Clip id, and packet ID (PID) are designated. The ref_to_SubPath_id indicates the SubPath_id, the ref_to_SubClip_entry_id indicates the Clip id, and the ref_to_stream PID_of_SubClip indicates the PID. This type is used when a plurality of clips are referred to by a SubPlayItem and when a plurality of elementary streams are referred to by each clip.

In this manner, when a PlayItem and at least one Sub Path played back in association with the PlayItem are provided, type (two types, i.e., type 1 and type 2) can be used for specifying one elementary stream from the clip referred to by the PlayItem and clips referred to by at least one SubPath. Type=1 indicates the clip (main clip) referred to by the Main Path and type=2 indicates a clip (sub clip) referred to by a Sub Path. '

Referring back to a description of the second STN_table () in Fig. 16, in the FOR loop of the audio stream ID (audio_stream_id), audio_stream_id is assigned from 0 to an audio elementary stream specified for each stream_entry(). Unlike the first STN_table() shown in Fig. 14, after Stream_entry() and Stream_attributes() for the first audio stream (decoded by a first audio decoder 75-1 of the playback apparatus 1 shown in Fig. 28), secondary_audio_present_flag indicating whether a secondary audio stream is defined is provided. If a secondary audio stream is defined, the secondary_audio_present_flag is set to be 1, and a secondary audio stream is defined in stream_entry() and stream_attributes() in the subsequent IF statement. If secondary_audio_present_flag is 1, two stream_entry() elements are defined in the FOR loop of the audio stream ID (audio_stream_id), and audio_stream_id is assigned from 0 to two audio elementary streams specified by the two stream_entry() elements. As in the case of the video streams, instead of the audio stream ID (audio_stream_id), the audio stream number (audio_stream_number) may be used, in which case, audio_stream_number is assigned from 1. That is, the number obtained by adding one to audio_stream_id is the audio_stream_number. The number is assigned from 1 since audio_stream_number is the audio stream number that can be seen by the user when switching sound.

It is possible that a Secondary Audio Stream, i.e., the secondary stream_entry(), is not entered. In this case, secondary_audio_present_flag is set to be 0, and only the primary audio stream (main audio stream) is entered. That is, if there is no secondary audio stream, the secondary stream_entry() is not entered. In this case, mixing of two audio streams is not performed.

In the FOR loop of the audio stream ID (audio_stream_id), two combinations of stream_entry() and stream attributes() are indicated. The first combination of stream_entry() and stream_attributes() is defined for an audio stream decoded by the first audio decoder 75-1 of the playback apparatus 1 shown in Fig. 28, and the second combination of stream_entry() and stream_attributes() entered when secondary_audio_present_flag is 1 is defined for an audio stream decoded by a second audio decoder 75-2 of the playback apparatus 1 shown in Fig. 28. That is, two audio streams can be defined.

In the FOR loop of the audio stream ID (audio_stream_id), stream_attributes() provides stream attribute information concerning at least one audio elementary stream specified for each stream_entry(). That is, in the stream_attributes(), stream attribute information concerning at least one audio elementary stream specified for each stream_entry() is indicated. More specifically, if secondary_audio_present_flag shown in Fig. 16 is 1, there are two audio elementary streams specified by the two stream_entry () elements, and thus, each stream_attributes () provides stream attribute information concerning the corresponding audio elementary stream. More specifically, in the FOR loop of the audio stream ID (audio_stream_id), the first stream_entry() defines the audio stream (main audio stream) decoded by the first audio decoder 75-1 of the playback apparatus 1 shown in Fig. 28, and then, stream attribute information concerning the specified audio elementary stream is given by the first stream_attributes(). Similarly, if secondary_audio_present_flag is 1, the second stream_entry() in the FOR loop of the audio stream ID (audio_stream_id) defines the audio stream decoded by the second audio decoder 75-2 of the playback apparatus 1 shown in Fig. 28, and then, stream attribute information concerning the specified audio elementary stream is given by the second stream_attributes() in the FOR loop of the audio stream ID (audio_stream_id).

The audio stream defined by the first stream_entry() in the FOR loop of the audio stream ID (audio_stream_id) is, for example, dubbed Japanese movie sound or the English movie sound. If secondary_audio_present_flag in the For loop of the audio stream ID (audio_stream_id) is 1, the audio stream defined by the second stream_entry () is, for example, movie sound of Director's Cut, which is used when director's commentaries are inserted into a predetermined portion of the AV stream file referred to by the Main Path.

That is, when combining two audio streams, secondary_audio_present_flag shown in Fig. 16, which indicates the presence of an audio stream to be combined with the primary (first) audio stream, is set to be 1 so that the stream_entry () and stream_attributes () of the secondary audio stream can be referred to.

More particularly, it is now assumed that one of the audio streams to be combined is set to be the audio stream referred to by the Main Path (main audio stream decoded by the first audio decoder 75-1 of the playback apparatus 1 shown in Fig. 28, which is discussed below), and the other audio stream is set to be the audio stream referred to by a Sub Path (secondary audio stream decoded by the second audio decoder 75-2 of the playback apparatus 1 shown in Fig. 28, which is discussed below). In this case, the type of first stream_entry() in Fig. 16 is set to be 1 (type=1), and stream attribute information concerning the specified audio elementary stream is given by stream_attributes(). Then, secondary_audio_present_flag in Fig. 16 is set to be 1, and the type of second stream_entry() is set to be 2 (type=2), and stream attribute information concerning the specified audio elementary stream is given by stream_attributes().

If two audio streams referred to by Sub Paths are combined, the type of first stream_entry() in the FOR loop of the audio stream ID (audio_stream_id) in Fig. 16 is set to be 2 (type=2), and stream attribute information concerning the specified audio elementary stream is given by stream_attributes. Then, secondary_audio_present_flag in the FOR loop of the audio stream ID (audio_stream_id) in Fig. 16 is set to be 1, and the type of second stream_entry () is set to be 2 (type=2). Then, stream attribute information concerning the specified audio elementary stream is given by stream_attributes(). That is, if audio streams to be combined are audio streams referred to by Sub Paths, the type of stream_entry() in Fig. 17 is 2 (type=2).

As described above, since audio streams (or video streams) can be specified by Sub Paths and added, an audio stream (or video stream) stored in a disc and an audio stream (or video stream) obtained by being downloaded or stored in a local storage can be played back in synchronization with each other. Alternatively, audio streams (or video streams) obtained by being downloaded can be played back in synchronization with a video stream (or audio stream) stored in a disc or obtained from an external source.

If a PlayList is obtained, a video stream (or audio stream) specified by the Main Path can be played back in synchronization with two audio streams (or video streams) specified by Sub Paths.

If only an audio stream referred to by the Main Path or only an audio stream referred to by a Sub Path is played back, secondary_audio_present_flag in Fig. 16 is set to be 0. Then, if the audio stream is referred to by the Main Path, the type of first stream_entry() is set to be 1 (type=1), and if the audio stream is referred to by the Sub Path, the type of first stream_entry() is 2 (type=2). Stream attribute information concerning the specified audio elementary stream is then given by stream_attributes(). That is, only one audio stream can be played back without being mixed.

In this manner, in Fig. 16, stream_entry () and stream_attributes can be inserted for two audio elementary streams. That is, in the FOR statement of the audio stream, by using two stream_entry() elements, the corresponding stream_attributes() can be entered.

That is, in the second STN_table() shown in Fig. 16, the two stream_entry() elements of the audio stream ID (audio_stream_id) define the corresponding stream_attributes(). Accordingly, one combination of stream_entry() and stream_attributes() referred to by the Main Path and the other combination of stream_entry() and stream_attributes() referred to by a Sub Path or by two Sub Paths can be entered for one audio_stream_id. Thus, two audio streams, which are played back simultaneously, can be entered.

The video stream ID (video_stream_id), subtitle stream ID (PG_txtST_stream_id), and graphics stream ID (IG_stream_id) in Fig. 16 are similar to those in Fig. 14.

The syntax of stream_attribute() is discussed below with reference to Fig. 18.

The length is a 16-bit unsigned integer indicating the number of bytes from the length field to the end of stream_attribute().

The stream_coding_type indicates the coding type of elementary stream, as shown in Fig. 19. The coding types of elementary streams include MPEG-2 video stream, HDMV LPCM audio, Dolby AC-3 audio, dts audio, Presentation graphics stream, Interactive graphics stream, and Text subtitle stream. The stream_coding_type may be further extended so that video streams compressed by other types of coding methods, such as MPEG4 AVC and VC-1, can be handled.

The video_format indicates the video format of a video elementary stream, as shown in Fig. 20. The video formats of the video elementary streams include 480i, 576i, 480p, 1080i, 720p, and 1080p.

The frame_rate indicates the frame rate of a video elementary stream, as shown in Fig. 21. The frame rates of the video elementary streams include 24000/1001, 24, 25, 30000/1001, 50, and 60000/1001.

The aspect_ratio indicates the aspect ratio of a video elementary stream, as shown in Fig. 22. The aspect ratios of the video elementary streams include 4:3 display aspect ratio and 16:9 display aspect ratio.

The audio_presentation_type indicates the presentation type of an audio elementary stream, as shown in Fig. 23. The presentation types of audio elementary streams include single mono channel, dual mono channel, stereo (2-channel), and multi-channel.

The sampling_frequency indicates the sampling frequency of an audio elementary stream, as shown in Fig. 24. The sampling frequencies of the audio elementary streams include 48 kHz and 96 kHz.

The audio_language_code indicates the language code (for example, Japanese, Korean, or Chinese) of an audio elementary stream.

The PG_language_code indicates the language code (for example, Japanese, Korean, or Chinese) of a bitmap subtitle elementary stream.

The IG_language_code indicates the language code (for example, Japanese, Korean, or Chinese) of an interactive graphics elementary stream.

The textST_language_code indicates the language code (for example, Japanese, Korean, or Chinese) of a text subtitle elementary stream.

The character_code indicates the character code of a text subtitle elementary stream, as shown in Fig. 25. The character codes of text subtitle elementary streams include Unicode V1.1 (ISO 10646-1), Shift JIS (Japanese), KSC 5601-1987 including KSC 5653 for Roman character (Korean), GB18030-2000 (Chinese), GB2312 (Chinese), and BIG5 (Chinese).

A specific example of the syntax of stream_attribute () shown in Fig. 18 is described below with reference to Figs. 18, and 19 through 25.

If the coding type (stream_coding_type in Fig. 18) of the elementary stream is the MPEG2 video stream (Fig. 19), stream_attribute() includes the video format (Fig. 20), the frame rate (Fig. 21), and the aspect ratio (Fig. 22) of the elementary stream.

If the coding type (stream_coding_type in Fig. 18) of the elementary stream is the HDMV LPCM audio, Dolby AC-3 audio, or dts audio (Fig. 19), stream_attribute () includes the audio presentation type information (Fig. 23), the sampling frequency (Fig. 24), and the language code of the audio elementary stream.

If the coding type (stream_coding_type in Fig. 18) of the elementary stream is the Presentation graphics stream (Fig. 19), stream_attribute() includes the language code of the bitmap subtitle elementary stream.

If the coding type (stream_coding_type in Fig. 18) of the elementary stream is the Interactive graphics stream (Fig. 19), stream attribute() includes the language code of the interactive graphics elementary stream.

If the coding type (stream_coding_type in Fig. 18) of the elementary stream is the Text subtitle stream (Fig. 19), stream_attribute() includes the character code (Fig. 25) and the language code of the text subtitle elementary stream.

The attribute information is not restricted to the above-described types.

In this manner, if a PlayItem and at least one Sub Path played back in association with the PlayItem are provided, by referring to the clip referred to by the PlayItem and clips referred to by at least one Sub Path, attribute information concerning an elementary stream (one elementary stream if type=1 or type=2, and two elementary streams if type=3) specified by stream_entry() can be defined by stream_attribute().

By checking the attribute information (stream attribute()), the playback apparatus can determine whether it has a function of playing back the corresponding elementary stream. Also, by checking the attribute information, the playback apparatus can select the elementary streams in accordance with the initial information concerning the language set in the playback apparatus.

It is now assumed, for example, that the playback apparatus has a function of playing back bitmap subtitle elementary streams without a function of playing back text subtitle elementary streams. In this case, in response to an instruction to switch the languages from the user, the playback apparatus sequentially selects only bitmap subtitle elementary streams from the FOR loop of the subtitle stream ID (PG_txtST_stream_id) and plays back the selected elementary streams.

If the initial information concerning the language set in the playback apparatus is Japanese, in response to an instruction to switch from main audio to sub audio (i.e., secondary audio) or to mix main audio with secondary audio from the user, the playback apparatus sequentially selects only audio elementary stream whose language code is Japanese from the FOR loop of the audio stream ID (audio_stream_id) prepared as secondary audio and plays back the selected elementary streams.

If AV video streams and AV audio streams (movie) referred to by the Main Path are played back, in response to an instruction to switch sound and to mix an audio stream (director's or performs' commentaries) referred to by a Sub Path with an AV stream referred to by the Main Path from the user, the playback apparatus mixes (superimposes) the second audio stream referred to by the Sub Path with (on) the first audio stream referred to by the Main Path, and plays back the mixed audio streams together with the video streams.

In response to an instruction to switch sound and to mix two audio streams referred to by two Sub Paths (i.e., an instruction to play back audio streams corresponding to the audio IDs whose secondary_audio_present_flag shown in Fig. 16 is 1 and whose types shown in Fig. 17 are 2 in the syntaxes shown in Figs. 16 and 17), the playback apparatus mixes (superimposes) the two audio streams referred to by the two Sub Paths, and plays back the mixed streams together with the video stream.

As described above, by the provision of STN_table() in PlayItem(), if a PlayItem and at least one Sub Path played back in association with the PlayItem are provided, the user can select a stream to be played back from the clip referred to by the PlayItem and clips referred to by at least one Sub Path when switching sound or subtitles. Thus, interactive operations can be performed for streams or data files different from an AV stream to be played back.

Since one PlayList includes a plurality of SubPaths and each SubPath refers to a SubPlayItem, AV streams having high extensibility and high flexibility are implemented. That is, SubPlayItems can be added afterwards. For example, if a PlayList including a clip AV stream file referred to by the Main Path is replaced by a PlayList including the clip AV stream file and a new Sub Path, the user can select based on the new PlayList, not only from the clip AV stream file referred to by the Main Path, but also from the clip AV stream file referred to by the Sub Path. Thus, the AV streams have high extensibility.

If a PlayItem and at least two Sub Paths played back associated with the PlayItem are provided, the STN_table() in PlayItem() provides a mechanism for mixing clip audio streams referred to by two Sub Paths and playing back the mixed stream (Figs. 16 and 17). Accordingly, interactive operations can be performed when superimposing an audio stream (for example, dubbed Japanese movie sound) decoded by the first audio decoder 75-1 on an audio stream (for example, Director's Cut) decoded by the second audio decoder 75-2.

By the provision of STN_table () in PlayItem (), if a PlayItem and at least one SubPath played back in association with the PlayItem are provided, a mechanism for mixing a first audio stream referred to by the PlayItem with a second audio stream referred to by the SubPath can be provided. Thus, the user can perform interactive operations when superimposing an audio stream different from the main clip AV stream (such as a second audio stream, for example, the director's commentaries) on the audio stream referred to by the PlayItem.

A specific example is given with reference to Fig. 26. Fig. 26 illustrates an example of the stream number table indicating the relationship between sound numbers and subtitle numbers provided to users. The stream number table shown in Fig. 26 corresponds to the second STN_table and second Stream_entry() shown in Figs. 16 and 17, respectively.

In Fig. 26, the sound numbers are referred to as "A_SN" (Audio Stream Number) and the subtitle numbers are referred to as "S_SN" (SubPicture Stream Number). In Fig. 26, each of a plurality of clip AV audio streams referred to by the Main Path and each of a plurality of clip AV audio streams referred to by a Sub Path are provided with A_SN. That is, audio 2 referred to by the Main Path is assigned to A_SN=1, audio 1 referred to by the Sub Path is assigned to A_SN=2, and audio 3 referred to by the Main Path is assigned to A_SN=3.

A_SN is assigned such that audio streams referred to by the Sub Path can be mixed with each of the audio streams provided with A_SN=1 through A_SN=3. More specifically, a combination of audio 4 referred to by another Sub Path and audio 2 referred to by the Main Path is assigned to A_ SN=4. A combination of audio 4 and audio 1 referred to by the Sub Paths is assigned to A_SN=5. In other words, A_SN=4 is assigned for mixing an audio stream referred to by the Main Path and an audio stream referred to by the Sub Path, and A_SN=5 is assigned for mixing two audio streams referred to by the two Sub Paths.

Each of a plurality of sub-picture streams of the main clip AV stream referred to by the Main Path is provided with S_SN. More specifically, sub-picture 3 is assigned to S_SN=1, sub-picture 1 is assigned to S_SN=2, and sub-picture 2 is assigned to S_SN=3. In this case, as A_SN or S_SN indicates a smaller number, the corresponding audio signals or sub-picture stream is provided to the user with higher priority. That is, A_SN=1 is an audio stream played back as a default, and S_SN=1 is a sub-picture stream played back as a default.

More specifically, sound played back based on the initial information concerning the language set in the playback apparatus is audio 2 (Fig. 26), which is A_SN=1, and sound played back after being switched from audio 2 is audio 1, which is A_SN=2 (Fig. 26).

To provide such a stream number table, STN_table() in PlayItem() referred to by PlayList() should be configured such that "audio 1 or audio 3" and "sub-picture 1, sub-picture 2, or sub-picture 3" can be referred to by the Main Path of the PlayItem, and "audio 2 or audio 4" can be referred to by SubPath() played back in association with PlayItem (). Then, in STN_table () (Fig. 16), audio_stream_id=0 (A_SN=1) is assigned to audio 2 (type=1), audio_stream_id=1 (A_SN=2) is assigned to audio 1 (type=2), audio_stream_id=2 (A_SN=3) is assigned to audio 3 (type=1), audio_stream_id=3 (A_SN=4) is assigned to audio 2 and audio 4 (type=1 and type=2), and audio_stream_id=4 (A_SN=5) is assigned to audio 1 and audio 4 (type=2 and type=2).

In the case of audio_stream_id=3 (A_SN=4), by the first stream_entry() and first stream_attributes() corresponding to the audio stream ID in Fig. 16, the audio stream (type=1) referred to by the Main Path can be registered, and the subsequent secondary_audio_present flag is set to be 1. Then, by the second stream_entry() and second stream_attributes(), the audio stream (type=2) referred to by the Sub Path can be registered. In this case, by using audio_stream_id, a combination of an audio stream referred to by the Main Path and an audio stream referred to by the Sub Path can be defined. That is, inconsistent combinations are not defined. This enables the user to always select a combination of audio streams that can be played back. That is, in the example shown in Fig. 26, an inconsistent combination, such as audio 2 and audio 3, is not defined, and thus, the user can select combinations that can be played back.

As discussed above, in stream_entry () (Fig. 17) in STN_table () (Fig. 16) in PlayItem (), the first audio stream referred to by the Main Path and the second audio stream referred to by the Sub Path can be entered (registered) by the provision of secondary_audio_present_flag. Then, a combination of the first audio stream and the second audio stream can be provided by audio_stream_id so that two audio streams can be mixed and played back. That is, from a plurality of types of streams, streams of the same type (in this case, audio streams) can be mixed (superimposed or combined) and played back at the same time.

In the case of audio_stream_id=4 (A_SN=5), by the first stream_entry() and first stream_attributes() and the second stream_entry() and second stream_attributes() corresponding to the audio stream ID in Fig. 16, the audio streams (type=2) referred to by the two Sub Paths can be registered. In this case, by using audio_stream_id, a combination of the two audio streams referred to by the two Sub Paths can be defined. That is, inconsistent combinations are not defined. This enables the user to always select a combination of audio streams that can be played back.

As discussed above, in stream_entry() (Fig. 17) in STN_table() (Fig. 16) in PlayItem(), two audio streams referred to by two Sub Paths can be entered (registered). Then, a combination of two audio streams can be provided by audio_stream_id so that two audio streams can be mixed and played back. That is, from a plurality of types of streams, streams of the same type (in this case, audio streams) can be mixed (superimposed or combined) and played back at the same time.

Additionally, the user can select audio streams to be played back from a plurality of predetermined combinations of audio streams without the need to individually select two audio streams.

Also in the syntaxes shown in Figs. 14 and 15, a combination of a first audio stream referred to by the Main Path and a second audio stream referred to by a Sub Path in Fig. 26 (audio_stream_id=3 (A_SN=4)) can be defined. In this case, in stream_entry() shown in Fig. 14, the audio stream referred to by the Main Path and the audio stream referred to by the Sub Path are registered by type=3, and then, two stream_attributes() elements are inserted. In this manner, in stream_entry() (Fig. 15) of STN_table() (Fig. 14) in PlayItem(), the first audio stream referred to by the Main Path and the second audio stream referred to by the Sub Path can be entered (registered) (type=3), and by using audio-stream-id, the first audio stream and the second audio stream are combined so that the two audio streams can be mixed and played back. That is, from a plurality of types of streams, streams of the same type (in this case, audio streams) can be mixed (superimposed or combined) and played back at the same time.

A first example of the configuration of the playback apparatus 1 to which the present invention is applied is discussed. Fig. 27 is a block diagram illustrating an example of the configuration of the playback apparatus 1 to which the present invention is applied. The playback apparatus 1 is the playback apparatus 1 playing back a PlayList including the above-described Main Path and Sub Path.

A controller 21 executes a predetermined control program. Alternatively, the controller 21 controls a storage drive 22 to read a navigation program recorded on a recording medium 11 or on a local storage 24 and expands it into a memory 23 to execute the navigation program. Then, the controller 21 controls the overall operation of the playback apparatus 1. For example, the controller 21 can display a predetermined menu screen on an external display device when the recording medium 11 is installed in the playback apparatus 1.

The storage drive 22 reads data from the recording medium 11 under the control of the controller 21 and outputs the read data to the controller 21, the memory 23, or an AV decoder 26. If information read from the recording medium 11 is a navigation program or a PlayList, it is output to the controller 21 or the memory 23. If information read from the recording medium 11 is an AV stream or text data, information read from the storage drive 22 is output to the AV decoder 26.

The memory 23 stores data necessary for executing various processing jobs by the controller 21. The memory 23 includes various registers in which information necessary for playback processing by the playback apparatus 1 is set. The local storage 24 may be a HDD (Hard Disk Drive).

An Internet interface 25 is connected to a network 2 by wired or wireless means, and communicates with a server 3 via the network 2 under the control of the controller 21 to supply data downloaded from the server 3 to the local storage 24. In this case, data downloaded from the server 3 as content is, for example, data to update information, such as that shown in Fig. 5, recorded on the recording medium 11 installed in the playback apparatus 1. The local storage 24 can record therein the content downloaded from the server 3 via the network 2.

The AV decoder 26 decodes AV streams or text data supplied from the storage drive 22 or the local storage 24 and outputs the resulting video signal and audio signal to an external display device. In the display device, content (video or sound) recorded on the recording medium 11, for example, is output based on the signal decoded by the AV decoder 26.

An operation input unit 29, which includes an input device, such as buttons, keys, a touch panel, a jog dial, or a mouse, and a receiver that receives signals, such as infrared signals, sent from a predetermined remote commander, receives the input from the user and supplies it to the controller 21.

A drive 27 is connected to the controller 21 if necessary, and a removable medium 28 including a magnetic disk (including a flexible disk), an optical disc (including a CD-ROM (Compact Disk-Read Only Memory) and a DVD), a magneto-optical disk (including an MD (registered)), or a semiconductor memory, is installed in the drive 27.

Fig. 28 is a block diagram illustrating the AV decoder 26 of the playback apparatus 1 shown in Fig. 27.

In the example shown in Fig. 28, the controller 21 first reads out a PlayList file from the storage drive 22 or the local storage 24, and reads an AV stream or AV data from the local storage 24 or the recording medium 11, such as an HDD, Blu-ray disc^™, or a DVD, via the storage drive 22 on the basis of the information concerning the PlayList file. The user can give an instruction to switch sound or subtitles to the controller 21 by using the operation input unit 29. The controller 21 reads various setting information including initial information concerning the language set in the playback apparatus 1 from the register of the memory 23.

The PlayList file includes, not only information concerning a Main Path and information concerning Sub Paths, but also STN_table(). The controller 21 reads a main clip AV stream file referred to by a PlayItem contained in the PlayList file, a sub-clip AV stream file referred to by a SubPlayItem, and text subtitle data referred to by a SubPlayItem from the recording medium 11 or the local storage 24 via the storage drive 22. In this case, the main clip AV stream referred to by the PlayItem and the sub-clip AV stream file referred to by the SubPlayItem may be recorded on different recording media. For example, the main clip AV stream file may be recorded on the recording medium 11, and the corresponding sub-clip AV stream file may be supplied via the network 2 (not shown) and stored in the local storage 24, such as an HDD. By referring to various setting information set in the register of the memory 23, the controller 21 controls the playback apparatus 1 to select and play back elementary streams in accordance with the playback function of the playback apparatus 1 or to select and play back elementary streams in accordance with the initial information concerning the language set in the playback apparatus 1.

The AV decoder 26 includes buffers 51 through 54, PID filters 55 and 56, switches 57 through 59, a background decoder 71, an MPEG (Moving Picture Experts Group)2 video decoder 72, a presentation graphics decoder 73, an interactive graphics decoder 74, the first audio decoder 75-1, the second audio decoder 75-2, a Text-ST composition 76, a switch 77, a background plane generator 91, a video plane generator 92, a presentation graphics plane generator 93, an interactive graphics plane generator 94, a buffer 95, a video data processor 96, and mixing processors 97 and 98. The first and second audio decoders 75-1 and 75-2 decode audio streams. The first audio decoder 75-1 is used preferentially over the second audio decoder 75-2. More specifically, if there is only one audio stream to be decoded, it is decoded by the first audio decoder 75-1 regardless of whether the audio stream belongs to the main clip AV stream file or the sub-clip AV stream file. That is, the playback apparatus 1 includes two audio decoders (the first audio decoder 75-1 and the second audio decoder 75-2) for decoding two audio streams. Hereinafter, the first and second audio decoders 75-1 and 75-2 are simply referred to as the "audio decoder 75" unless they have to be distinguished from each other.

File data read by the controller 21 is demodulated by a demodulator, and the demodulated multiplexed streams are then subjected to error correction by an ECC decoder. A switch 32 then selects the multiplexed streams according to the stream types and supplies the streams to the corresponding buffers 51 through 54 under the control of the controller 21. More specifically, under the control of the controller 21, the switch 32 supplies background image data to the buffer 51, main clip AV stream file data to the buffer 52, sub-clip AV stream file data to the buffer 53, and text-ST data to the buffer 54. Then, the buffers 51 through 54 buffer the background image data, main clip AV stream file data, sub-clip AV stream file data, and text-ST data, respectively, therein.

The main clip AV stream file is a stream (for example, a transport stream) in which at least one stream of video, audio, bitmap subtitle (presentation graphics), and interactive graphics streams is multiplexed together with a video stream. The sub-clip AV stream file is a stream in which at least one stream of audio, bitmap subtitle (Presentation Graphics stream), and interactive graphics streams is multiplexed. The text subtitle data file (Text-ST) may be a multiplexed stream, such as a transport stream, but this is not essential.

When reading the main clip AV stream file, the sub-clip AV stream file, and the text subtitle data from the storage drive 22 (recording medium 11), they can be alternately read in a time-division manner. Alternatively, the sub-clip AV stream file or the text subtitle data may be entirely preloaded to the buffer 53 or 54, respectively, before reading the main clip.

The playback apparatus 1 reads those data from the recording medium 11 via the storage drive 22 to play back video, bitmap subtitle, interactive graphics, and audio.

More specifically, stream data read out from the buffer 52, which serves as the clip AV stream file read buffer, is output to the PID (packet ID) filter 55, which is disposed subsequent to the buffer 52 at a predetermined time. The PID filter 55 allocates streams contained in the main clip AV stream file to the corresponding elementary stream decoders according to the PIDs (packet IDs). More specifically, the PID filter 55 supplies video streams to the video decoder 72, presentation graphics streams to the switch 57, which supplies the graphics streams to the presentation graphics decoder 73, interactive graphics streams to the switch 58, which supplies the graphics streams to the interactive graphics decoder 74, and audio streams to the switch 59, which supplies the audio streams to the first audio decoder 75-1.

The presentation graphics streams are, for example, bitmap subtitle data, and the text subtitle data is, for example, text subtitle data.

Stream data read out from the buffer 53, which serves as the sub-clip AV stream file read buffer, is output to the PID (packet ID) filter 56, which is disposed subsequent to the buffer 53, at a predetermined time. The PID filter 56 allocates streams contained in the sub-clip AV stream file to the corresponding elementary stream decoders according to the PIDs (packet IDs). More specifically, the PID filter 56 supplies presentation graphics streams to the switch 57, which supplies the graphics streams to the presentation graphics decoder 73, interactive graphics streams to the switch 58, which supplies the graphics streams to the interactive graphics decoder 74, and sub-clip audio streams to the switch 59, which supplies the audio streams to the first audio decoder 75-1 and the second audio decoder 75-2.

Data read out from the buffer 51, which serves as the background image data buffer, is supplied to the background decoder 71 at a predetermined time. The background decoder 71 decodes the background image data, and then supplies the decoded data to the background plane generator 91.

The video streams allocated by the PID filter 55 are supplied to the video decoder 72, which is disposed subsequent to the PID filter 55. The video decoder 72 decodes the video streams and supplies the decoded video streams to the video plane generator 92.

The switch 57 selects one of the presentation graphics streams contained in the main clip AV stream file supplied from the PID filter 55 and the presentation graphics streams contained in the sub-clip AV stream file supplied from the PID filter 56, and supplies the selected presentation graphics streams to the presentation graphics decoder 73, which is disposed subsequent to the switch 57. The presentation graphics decoder 73 decodes the presentation graphics streams and supplies them to the switch 77, which further supplies them to the presentation graphics plane generator 93.

The switch 58 selects one of the interactive graphics streams contained in the main clip AV stream file supplied from the PID filter 55 and the interactive graphics streams contained in the sub-clip AV stream file supplied from the PID filter 56, and supplies the selected interactive graphics streams to the interactive graphics stream decoder 74, which is disposed subsequent to the switch 58. That is, the interactive graphics streams simultaneously input into the interactive graphics decoder 74 are streams separated from the main clip AV stream file or the sub-clip AV stream file. The interactive graphics decoder 74 decodes the interactive graphics streams, and supplies the decoded streams to the interactive graphics plane generator 94.

The switch 59 selects one of the audio streams contained in the main clip AV stream file supplied from the PID filter 55 and the audio streams contained in the sub-clip AV stream file supplied from the PID filter 56 and supplies the selected audio streams to the first audio decoder 75-1 or the second audio decoder 75-2. The audio streams simultaneously input into the first audio decoder 75-1 are streams separated from the main clip AV stream file or the sub-clip AV stream file. The audio streams simultaneously input into the second audio decoder 75-2 are streams separated from the sub-clip AV stream file.

More specifically, the switch 59 selects the audio streams according to the type in stream_entry() shown in Fig. 15 or 17. If type=1, the switch 59 selects the audio streams contained in the main clip AV stream file supplied from the PID filter 55 and supplies them to the first audio decoder 75-1. If type=2, the switch 59 selects the audio streams contained in the sub-clip AV stream file supplied from the PID filter 56 and supplies them to the first audio decoder 75-1 or the second audio decoder 75-2.

If an instruction to mix an audio stream contained in a sub-clip AV stream file with an audio stream contained in the main clip AV stream file is given (i.e., if the type in stream_entry() of the first STN_table shown in Fig. 14 is 3 (type=3), or if secondary_audio_present_flag in stream_entry of the second STN_table shown in Fig. 16 is 1 and the type in the second stream_entry() is 2 (type=2)), the switch 59 supplies the audio stream contained in the main clip AV stream file supplied from the PID filter 55 to the first audio decoder 75-1 and supplies the audio stream contained in the sub-clip AV stream file supplied from the PID filter 56 to the second audio decoder 75-2.

If an instruction to mix two audio streams contained in sub-clip AV stream files is given (i.e., if, in Fig. 16, secondary_audio_present_flag is 1, and the type of first stream_entry () is 2 (type=2) and the type of second stream_entry () is 2 (type=2)), the switch 59 supplies the corresponding audio stream contained in one sub-clip AV stream file supplied from the PID filter 56 to the first audio decoder 75-1 if that audio stream corresponds to the audio stream defined by the first stream_entry() in the FOR loop of the audio stream ID (audio_stream_id) shown in Fig. 16, and also supplies the corresponding audio stream contained in the other sub-clip AV stream file supplied from the PID filter 56 to the second audio decoder 75-2 if that audio stream corresponds to the audio stream defined by the second stream_entry() in the FOR loop of the audio stream ID (audio_stream_id) shown in Fig. 16.

That is, audio streams read from the buffer 52, which serves as the main clip AV stream file read buffer (when type=1 or type=3 in Fig. 15 or when type=1 in Fig. 17), and audio streams read from the buffer 53, which serves as the sub-clip AV stream file read buffer (when type=2 or type=3 in Fig. 15 or when type=2 in Fig. 17) are supplied to the first audio decoder 75-1. Only audio streams read from the buffer 53 are supplied to the second audio decoder 75-2.

The first audio decoder 75-1 decodes the audio streams and supplies the decoded audio streams to the mixing processor 101. The second audio decoder 75-2 decodes the audio streams and supplies the decoded audio streams to the mixing processor 101.

If two audio streams are superimposed and played back, the audio stream decoded by the first audio decoder 75-1 and the audio stream decoded by the second audio decoder 75-2 are supplied to the mixing processor 101.

For example, if the first audio stream referred to by the Main Path and the second audio stream referred to by a Sub Path are superimposed and played back (if type=3 in Fig. 15, or if, in Fig. 16, secondary_audio_present_flag is 1 and the type of first stream_entry() is 1 (type=1) and the type of second stream_entry() is 2 (type=2)), the audio stream contained in the main clip AV stream file decoded by the first audio decoder 75-1 and the audio stream contained in the sub-clip AV stream file decoded by the second audio decoder 75-2 are supplied to the mixing processor 101.

If two audio streams referred to by Sub Paths are superimposed and played back (if, in Fig. 16, secondary audio present flag is 1, and if the types of the first stream_entry() and the second stream entry() are 2 (type=2)), the audio stream contained in one sub-clip AV stream file decoded by the first audio decoder 75-1 (type=2 and defined by the first stream_entry() in Fig. 16) and the audio stream contained in the other sub-clip AV stream file decoded by the second audio decoder 75-2 (type=2 and defined by the second stream_entry() when secondary_audio_present_flag is 1 in Fig. 16) are supplied to the mixing processor 101.

The mixing processor 101 mixes (superimposes) the audio data from the first audio decoder 75-1 and the audio data from the second audio decoder 75-2, and supplies the resulting data to the mixing processor 97. In this embodiment, mixing (superimposing) of the audio data output from the first audio decoder 75-1 and the audio data output from the second audio decoder 75-2 is also referred to as "combining". That is, to combine two audio data is to mix two audio data.

Sound data selected by the switch 32 is supplied to the buffer 95 and is buffered therein. The buffer 95 supplies the sound data to the mixing processor 97 at a predetermined time. The sound data is, for example, effect sound that can be selected from a menu. The mixing processor 97 mixes (superimposes or combines) the audio data mixed by the mixing processor 101 (audio data obtained by mixing the audio data output from the first audio decoder 75-2 with the audio data output from the second audio decoder 75-2) with the sound data supplied from the buffer 95, and outputs the resulting data as an audio signal.

Data read from the buffer 54, which serves as the text subtitle read buffer, is output to the text subtitle composition (decoder) 76, which is disposed subsequent to the buffer 54, at a predetermined time. The text subtitle composition 76 decodes the text-ST data and supplies the decoded data to the switch 77.

The switch 77 selects one of the presentation graphics streams decoded by the presentation graphics decoder 73 and the Text-ST (text subtitle data) decoded by the text subtitle composition 76, and supplies the selected data to the presentation graphics plane generator 93. That is, subtitle images simultaneously supplied to the presentation graphics plane generator 93 are those output from the presentation graphics decoder 73 or from the text subtitle (Text-ST) composition 76. Presentation graphics streams simultaneously input into the presentation graphics decoder 73 are streams separated from the main clip AV stream file or from the sub-clip AV stream file (selected by the switch 57). Accordingly, the subtitle images simultaneously input into the presentation graphics plane generator 93 are presentation graphics streams from the main clip AV stream file, presentation graphics streams from a sub-clip AV stream file, or text subtitle data.

The background plane generator 91 generates a background plane, which serves as, for example, a wallpaper image when a video image is displayed by reducing the size thereof, on the basis of the background image data supplied from the background decoder 71, and supplies the generated background plane to the video data processor 96. The video plane generator 92 generates a video plane based on the video data supplied from the video decoder 72, and supplies the gene